Imaging sensor and imaging device

ABSTRACT

An imaging sensor includes a color filter, and DBPF that has a transmission characteristic in a visible-light band, a blocking characteristic in a first wavelength band adjacent to a long-wavelength side of the visible-light band, and a transmission characteristic in a second wavelength band that is a part of the first wavelength band. A transmission characteristic of DBPF and a transmission characteristic of each filter part of the color filter are set in such a manner that the second wavelength band of DBPF is included in a third wavelength band that is a wavelength band in which transmittance of the filter parts in colors is approximate to each other on a long-wavelength side of the visible-light band and a fourth wavelength band that is a wavelength band in which a filter part for infrared light has a transmission characteristic.

This application is a continuation of U.S. patent application Ser. No.15/320,234 filed on Dec. 19, 2016, which is a National Stage Entry ofInternational Application No. PCT/JP2015/068280 filed on Jun. 24, 2015,which claims priority from Japanese Patent Application No. 2014-220183filed on Oct. 29, 2014, and Japanese Patent Application No. 2014-129636filed on Jun. 24, 2014. The entire contents of each of theabove-identified documents are incorporated by reference herein.

TECHNICAL FIELD

The present invention relates to an imaging sensor and an imaging devicethat perform both of photographing with visible light and photographingwith infrared light.

BACKGROUND ART

Conventionally, in an imaging device such as a monitoring camera thatcontinuously performs photographing night and day (hereinafter, simplyreferred to as imaging device), infrared light is detected andphotographing is performed at night. A photodiode that is alight-receiving unit of an imaging sensor such as a CCD sensor or a CMOSsensor can receive light of up to a near-infrared wavelength band around1300 nm. Thus, an imaging device using these imaging sensors can performphotographing of up to an infrared band in principle.

Note that a wavelength band of light with a high luminosity factor of ahuman is 400 nm to 700 nm. Thus, when near-infrared light is detected inan imaging sensor, redness is increased in a video seen with a humaneye. Thus, in photographing in daytime or in a bright indoor place, itis preferable that an infrared cut filter to block light in an infraredband is provided in front of the imaging sensor and light with awavelength being 700 nm or longer is removed in order to adjustsensitivity of the imaging sensor to a luminosity factor of a human. Onthe other hand, in photographing at night or in a dark place, it isnecessary to perform photographing without providing the infrared cutfilter.

As such an imaging device, an imaging device in whichattachment/detachment of an infrared cut filter is performed manually oran imaging device in which an infrared cut filter is inserted/removedautomatically is conventionally known. Moreover, in Patent Literature 1,an imaging device in which the above-described insertion/removal of aninfrared cut filter is not necessary is disclosed.

Thus, an optical filter that has a transmission characteristic in avisible-light band, a blocking characteristic in a first wavelength bandadjacent to a long-wavelength side of the visible-light band, and atransmission characteristic in a second wavelength band that is a partof the first wavelength band has been proposed (see, for example, PatentLiterature 1). According to this filter, light can be transmitted inboth of the visible-light band and the second wavelength band that isaway from the visible-light band on the long-wavelength side, that is,on an infrared side of the visible-light band. For example, the secondwavelength band overlaps with a wavelength band of infrared illuminationand this filter is an optical filter that makes it possible to performboth of visible light photographing and infrared light photographing atnight with infrared-light illumination. In the following, an opticalfilter that transmits light in a visible-light band and a secondwavelength band on an infrared side and that blocks light in the otherwavelength band in the above-described manner is referred to as a doublebandpass filter (DBPF).

CITATION LIST Patent Literature

Patent Literature 1: Japanese Patent No. 5009395

SUMMARY OF INVENTION Technical Problem

Incidentally, in DBPF in Patent Literature 1, light in a secondwavelength band included in an infrared (near-infrared) wavelength band(relatively narrow wavelength band included in infrared wavelength band)is not constantly blocked and the light passes. That is, unlike a casewhere an infrared cut filter that cuts a long-wavelength side of avisible-light band is used, there is more than a little influence ofinfrared light that passes through the second wavelength band inphotographing in the visible-light band.

A color filter is used in an imaging sensor that performs colorphotographing as photographing in the visible-light band. In the colorfilter, red, green, and blue regions (filter part) are arranged in apredetermined pattern in a manner corresponding to each pixel of theimaging sensor. Each of these color regions basically has a peak oftransmittance of light in a wavelength band of each color and limits(block) transmission of light in a wavelength band of a different color.

However, on the long-wavelength side of the visible-light band, light isbasically transmitted although light transmittance is differentdepending on a region of each color and a wavelength. Thus, when aninfrared cut filter is used, there is no problem since light on thelong-wavelength side of the visible-light band is cut. However, wheninfrared light is transmitted in the second wavelength band on theinfrared side in a case of DBPF described above, this infrared lightpasses through a color filter, reaches a photodiode (light-receivingelement) of an imaging sensor, and increases a generation amount ofelectrons by a photoelectric effect in the photo diode.

Here, for example, in a case where both of color photographing withvisible light and photographing with infrared-light illumination areperformed, a region for infrared light which region has a peak of lighttransmittance in the above-described second wavelength band (infraredregion) is provided in a color filter in which red, green, and blueregions are arranged in a predetermined pattern. That is, an array(pattern) in the color filter includes four regions of red R, green G,blue B, and infrared IR. In this case, since the region for infraredlight blocks light in the visible-light band and mainly transmits lightin the second wavelength band, it is considered that an infraredcomponent is removed from red, green, and blue image signals byutilization of an image signal of infrared light which signal is outputfrom an imaging sensor that receives light that passes through theregion for infrared light in the color filter. However, due to suchsignal processing, it is difficult to perform color reproduction that issubstantially equivalent to that in color photographing in a case wherean infrared cut filter is used.

The present invention is provided in view of the forgoing and is toprovide an imaging sensor and an imaging device that can improve colorreproducibility invisible light photographing in a case where DBPF isused instead of an infrared cut filter.

Solution to Problem

In order to solve the above problem, an imaging sensor of the presentinvention includes: an imaging sensor main body in which alight-receiving element is arranged in each pixel; and a filter providedon the imaging sensor main body, wherein on the filter, a plurality ofkinds of filter regions with different spectral transmissioncharacteristics is arranged in a predetermined array in a mannercorresponding to an arrangement of the pixel of the imaging sensor mainbody, the plurality of kinds of filter regions has different spectraltransmission characteristics each of which corresponds to a wavelengthin a visible-light band, and each of the plurality of kinds of filterregions has an infrared-light transmission wavelength band, which passeslight, on a long-wavelength side of the visible-light band and has alight-blocking wavelength band, which blocks light, between thevisible-light band and the infrared-light transmission wavelength bandin a mutually similar manner.

According to such a configuration, for example, in a camera that outputsan image signal in a visible-light band corresponding to each of red,green, and blue (RGB) and an image signal in infrared (IR) on along-wavelength side of the visible-light band, by using a filterincluding, for example, four kinds of filter regions arranged in amanner corresponding to each pixel, it is possible to remove a componenton the long-wavelength side of the visible-light band, which componentis included in each filter region, from the image signal in thevisible-light band at performance close to that of when an infrared cutfilter is used.

That is, in a color filter of a camera that uses an infrared cut filter,a transmission characteristic corresponding to a wavelength on along-wavelength side of visible light varies depending on a color ofeach filter part and a color varies due to the different in thetransmission characteristic when an uniformly-set infrared component isremoved in the filter part in each color. However, when a part, in whicha transmission characteristic of a wavelength varies on thelong-wavelength side of the visible-light band, in each filter region isblocked by a light-blocking wavelength band, it is possible to control avariation in color even when the uniformly-set infrared component isremoved. Note that, for example, in a case where colors are four colorsof R, G, B, and IR, when there are four kinds of filter regions withdifferent transmission characteristics corresponding to wavelengths inthe visible-light band, it is possible to calculate R, G, B, and IR andto output an image signal in the visible-light band, from which signalan infrared component is removed, and an infrared image signal.

In the configuration of the present invention, it is preferable that thefilter includes an optical filter that transmits light in thevisible-light band having a first wavelength band that blocks light in amanner adjacent to the long-wavelength side of the visible-light bandand that includes the light-blocking wavelength band, and a secondwavelength band as the infrared-light transmission wavelength band thattransmits light in a manner adjacent to a long-wavelength side of thelight-blocking wavelength band in a part away from the visible-lightband in the first wavelength band, and a color filter including aplurality of kinds of filter parts which has different spectraltransmission characteristics corresponding to the wavelengths in thevisible-light band having a third wavelength band of which transmittanceis approximate to each other on the long-wavelength side of thevisible-light band, and that corresponds to the plurality of kinds offilter regions, and a spectral transmission characteristic of theoptical filter and the spectral transmission characteristic of each ofthe filter parts of the color filter are set in such a manner that thesecond wavelength band of the optical filter is included in the thirdwavelength band.

According to such a configuration, an optical filter is used instead ofan infrared cut filter in photographing of a visible light image. Sinceinfrared light that passes through a second wavelength band of anoptical filter reaches a light-receiving element of each pixel of animaging sensor main body and increases the number of electrons generatedby a photoelectric effect, it is necessary to control an influence ofthe infrared light, which passes through the optical filter, byprocessing an image signal.

In this case, for example, when the colors are red, green, and blue,transmittance of a red filter part is substantially the maximum in aboundary part between a visible-light band and an infrared band and thetransmittance is substantially the maximum on a long-wavelength sidethereof. Also, in each of green and blue filter parts, transmittance islow in the boundary between the visible-light band and the infraredband. However, the transmittance becomes higher as a wavelength becomeslonger than the visible-light band and finally becomes substantially themaximum. The transmittance is also substantially the maximum on along-wavelength side of the wavelength at which the transmittancebecomes substantially the maximum.

In this case, in an infrared band adjacent to the visible-light band,that is, a region in which a wavelength of an infrared band is short,the transmittance of the blue and green filter parts is greatlydifferent from that of the red filter part with the maximumtransmittance and the transmittance varies between blue and green. Also,in each of the blue and green filter parts, transmittance trends becomehigher toward the long-wavelength side and the transmittance variesdepending on a wavelength until the transmittance becomes substantiallythe maximum. Thus, in a case where a wavelength band of this part isincluded in the second wavelength band, it is difficult to preventinfrared light that passes through a second wavelength low region frominfluencing a visible image. That is, in a case of correcting red,green, and blue image signals based on light, which passes through thered, green, and blue filter parts, by using an infrared image signalbased on light that passes through the infrared filter part, it is notpossible to calculate signals, which are subtracted from the red, green,and blue image signals, from the infrared image signal based on thelight that passes through the infrared filter and it becomes difficultto improve color reproducibility.

Since the transmittance of the red, green, and blue filter parts becomessubstantially the maximum in the second wavelength band of the opticalfilter part, when the second wavelength band is included in a thirdwavelength band in which the transmittance of the red, green, and bluefilter parts is substantially approximate to each other, transmittanceof light that passes through the second wavelength band and that passesthrough the red, green, and blue filter parts becomes substantially thesame in the red, green, and blue filter parts. That is, a wavelengthband in which the transmittance varies greatly in the red, green, andblue filter parts overlaps with a wavelength band with a blockingcharacteristic between the visible-light band, which has a transmissioncharacteristic, and the second wavelength band in the optical filter.

Accordingly, light in the wavelength band in which transmittance variesgreatly depending on a color of a filter part cannot pass through theoptical filter. On an infrared side, light in a wavelength band in whichthe transmittance is approximate in the red, green, and blue filterparts can pass through the optical filter.

Note that the filter parts are not limited to red, green, blue, andinfrared and a filter part in a different color may be used. Forexample, color in the visible-light band other than red, green, andblue, white (clear (C) or white (W)) to transmit light in substantiallya whole wavelength band in the visible-light band, or the like may beused instead of infrared. Also, different colors may be used as red,green, and blue.

Also, white may be one in which transmittance of light is lowered in analmost similar manner in the whole visible-light band.

In the configuration of the present invention, in the third wavelengthband, it is preferable that a difference in transmittance of the filterparts in the colors is 20% or smaller in the transmittance.

According to such a configuration, it becomes possible to control aninfluence on a visible image by the infrared light passing through thesecond wavelength band of the optical filter as described above and itis possible to improve color reproducibility. Basically, it ispreferable that transmittance of the filter parts in the colors issubstantially identical to each other. However, the above-describedimage processing of controlling an influence of infrared light can beperformed when a difference is 10% or smaller in the transmittance. Notethat a state in which a difference in transmittance becomes 10% orsmaller in transmittance is a case where a difference in transmittanceof when the lowest transmittance (%) is subtracted from the highesttransmittance in the filter parts in the colors in the second wavelengthband becomes equal to or smaller than 20%. Note that a difference in thetransmittance is preferably 10% or smaller.

Also, in the configuration of the present invention, four or more kindsof the filter regions each of which has a transmission characteristic ina limited wavelength band corresponding to each color in thevisible-light band, a transmission characteristic in substantially awhole wavelength band in the visible-light band, or a blockingcharacteristic in substantially the whole wavelength band in thevisible-light band are preferably included.

According to such a configuration, in addition to red, green, and blue,an infrared filter having a blocking characteristic in substantially thewhole wavelength band in the visible-light band, or a white (clear)filter having a transmission characteristic in substantially the wholewavelength band in the visible-light band can be used. Also, a filter incolor other than red, green, and blue can be used. Also, red, green andblue filters may have different colors. Accordingly, a degree of freedomin designing of a color filter becomes high and various kinds ofdesigning become possible.

Also, in the configuration of the present invention, it is preferablethat the color filter includes four or more kinds of the filter partscorresponding to four or more kinds of different colors, one kind of thefilter parts has a blocking characteristic in substantially the wholevisible-light band and has a transmission characteristic in a fourthwavelength band on the long-wavelength side of the visible-light band,and a transmission characteristic of the optical filter and atransmission characteristic of each of the filter parts of the colorfilter are set in such a manner that the second wavelength band of theoptical filter is included in the third wavelength band and the fourthwavelength band.

According to such a configuration, it is possible to calculate aninfrared component by using an infrared filter part having a blockingcharacteristic in substantially the whole wavelength region in thevisible-light band. Also, the fourth wavelength band that is awavelength band, which has a transmission characteristic, of theinfrared filter part has a part that overlaps with the third wavelengthband. The second wavelength band of the optical filter is included inboth of the third wavelength band and the fourth wavelength band.Accordingly, in a case of eliminating an influence of infrared lightthat passes through the second wavelength band by image processing, itbecomes possible to calculate signals, which are subtracted from red,green and blue image signals, by using an infrared image signal that isbased on light passing through the infrared filter part and it becomespossible to improve color reproducibility by control ling an influenceof the infrared light that passes through the second wavelength band ofthe optical filter. That is, when it is assumed that the same quantityof infrared light in the same wavelength band enters each pixel,quantity of light that passes through the second wavelength band andpasses through the red, green, blue, and infrared filter parts does notvary greatly depending on a filter part and becomes substantially thesame. Thus, in this case, it is possible to remove light quantity basedon the infrared light by subtracting quantity of the light that passesthrough the infrared filter part from quantity of the light that passesthrough the red, green and blue filter parts.

Also, in the configuration of the present invention, it is assumed thatinfrared illumination is used in imaging of an infrared image, andsetting is preferably performed in such a manner that a fifth wavelengthband, which is a wavelength band of infrared light emitted from theinfrared illumination, is included in the third wavelength band and thefourth wavelength band and that the second wavelength band of theoptical filter substantially overlaps with the fifth wavelength band.

According to such a configuration, it is possible to make a wavelengthband of infrared light of the infrared illumination and the secondwavelength band of the optical filter substantially the same. Also, theinfrared light of the infrared illumination can be efficiently usedmainly in a case where photographing of an infrared image is performedat night by utilization of the infrared illumination. Also, in a casewhere a wavelength band of infrared light of the infrared illuminationis narrow, it is possible to narrow down the second wavelength band ofthe optical filter in accordance therewith. In this case, it is possibleto reduce an influence of infrared light, which passes through thesecond wavelength band, with respect to a visible image.

Also, in the configuration of the present invention, it is preferablethat in the color filter, four each of four kinds of the filter parts ofred, blue, and green with transmission characteristics in limitedwavelength bands corresponding to the colors in the visible-light bandand of infrared with a blocking characteristic in substantially a wholewavelength band in the visible-light band are arranged in a basic arraywith four rows and four columns, the same kinds of filter parts arearranged separately in such a manner as not to be adjacent to each otherin a row direction and a column direction, one each of the red, blue,green and infrared filter parts are arranged in each column, and twoeach of two kinds of the filter parts among the red, blue, green, andinfrared filter parts are arranged in every other row.

According to such a configuration, in the color filter, the red, blue,green, and infrared filter parts are arranged in substantially an evenmanner. In this case, it is not possible to use high sensitivity of ahuman eye with respect to green. However, the filter parts in the colorsare arranged in substantially an even manner and interpolationprocessing becomes easy. Also, in this color filter, one each of thered, blue, green, and infrared filter parts are arranged in each columnand two each of two kinds of the filter parts among the red, blue,green, and infrared filter parts are arranged in every other row. Thus,resolution in the row direction is higher than that in the columndirection. Note that it is preferable that the row direction is ahorizontal direction and the column direction is a vertical direction.

Also, in the configuration of the present invention, it is preferablethat in the color filter, eight green filter parts, four red filterparts, two blue filter parts and infrared filter parts among four kindsof the filter parts of red, blue, and green with transmissioncharacteristics in limited wavelength bands corresponding to colors inthe visible-light band and of infrared with a blocking characteristic insubstantially a whole wavelength band in the visible-light band arearranged in a basic array with four rows and four columns, and the samekinds of filter parts are arranged separately in such a manner as not tobe adjacent to each other in a row direction and a column direction.

According to such a configuration, resolution of blue is lower than thatin a color filter in the Bayer array with no infrared filter part.However, resolution of green and red can be held. In this case, there ispriority on green and luminance based on green.

Also, in the configuration of the present invention, it is preferablethat in the color filter, eight green filter parts, four infrared filterparts, two red filter parts and blue filter parts among four kinds ofthe filter parts of red, blue, and green with transmissioncharacteristics in limited wavelength bands corresponding to colors inthe visible-light band and of infrared with a blocking characteristic insubstantially a whole wavelength band in the visible-light band arearranged in a basic array with four rows and four columns, and the samekinds of filter parts are arranged separately in such a manner as not tobe adjacent to each other in a row direction and a column direction.

According to such a configuration, resolution of red and blue is lowerthan that in a color filter in the Bayer array with no infrared filterpart. However, resolution of green can be held. Also, resolution ofinfrared becomes higher than that of blue or red and it is possible tosecure resolution of an infrared image.

Also, in the configuration of the present invention, a signalseparating/outputting device to separate a signal, which is sequentiallyinput from each pixel of the imaging sensor main body, into an imagesignal in the visible-light band, from which signal a signal in awavelength band on the long-wavelength side of the visible-light band isremoved, and an infrared image signal on the long-wavelength side of thevisible-light band and to perform an output thereof is preferablyincluded.

According to such a configuration, since it is possible to output animage signal similar to that from a conventional imaging sensor tooutput an image signal in a visible-light band, it is possible to use animaging sensor of the present invention only by adding a processingcircuit of an infrared image signal without changing a processingcircuit of an image signal in and after the imaging sensor. Accordingly,it becomes easy to introduce the imaging sensor of the presentinvention. For example, it is possible to easily introduce aphotographing function of an infrared image at low cost into asmartphone, an electronic camera, or a different device into which acamera mechanism is embedded.

An imaging device of the present invention preferably includes: theimaging sensor; an optical system including a lens to form an image onthe imaging sensor; and a signal processing device that can process asignal output from the imaging sensor and can output a visible imagesignal and an infrared image signal.

According to such a configuration, an effect of the above-describedimaging device can be acquired. Thus, it is possible for the signalprocessing device to perform processing of eliminating an influence ofinfrared light, which passes through a second wavelength band of anoptical filter, in a visible image.

Also, an imaging device of the present invention includes: an imagingsensor including an imaging sensor main body in which a light-receivingelement is arranged in each pixel and a color filter in which aplurality of kinds of filter parts is arranged in a predetermined arrayin a manner corresponding to an arrangement of the pixel of the imagingsensor main body, the plurality of kinds of filter parts havingdifferent transmission characteristics corresponding to wavelengths in avisible-light band; an optical system including a lens to form an imageon the imaging sensor; an optical filter that is provided in the opticalsystem and that has a transmission characteristic in the visible-lightband, a blocking characteristic in a first wavelength band adjacent to along-wavelength side of the visible-light band, and a transmissioncharacteristic in a second wavelength band that is a part of the firstwavelength band; and a signal processing device that can process asignal output from the imaging sensor and can output a visible imagesignal and an infrared image signal, wherein a transmissioncharacteristic of the optical filter and the transmission characteristicof each of the filter parts of the color filter are set in such a mannerthat the second wavelength band of the optical filter is included in athird wavelength band that is a wavelength band in which transmittanceof the filter parts in colors is approximate to each other on thelong-wavelength side of the visible-light band.

According to such a configuration, in the above-described imagingdevice, an optical filter is provided not in an imaging sensor but in anoptical system. However, an optical characteristic of the optical filteror a color filter is similar to that of the above-described imagingdevice and an effect similar to that of the above-described imagingdevice can be acquired.

In the configuration of the imaging device of the present invention, itis preferable that a difference in the transmittance of the filter partsin the colors is 10% or smaller in the transmittance in the thirdwavelength band.

According to such a configuration, it becomes possible to control aninfluence on a visible image by the infrared light passing through thesecond wavelength band of the optical filter as described above and itis possible to improve color reproducibility. Basically, it ispreferable that transmittance of the filter parts in the colors issubstantially identical to each other. However, the above-describedimage processing of controlling an influence of infrared light can beperformed when a difference is 10% or smaller in the transmittance.

Also, in the configuration of the imaging device of the presentinvention, the color filter preferably includes four or more kinds ofthe filter parts each of which has a transmission characteristic in alimited wavelength band corresponding to each color in the visible-lightband, a transmission characteristic in substantially a whole wavelengthband in the visible-light band, or a blocking characteristic insubstantially the whole wavelength band in the visible-light band.

According to such a configuration, in addition to red, green, and blue,an infrared filter having a blocking characteristic in substantially thewhole wavelength band in the visible-light band, or a white (clear)filter having a transmission characteristic in substantially the wholewavelength band in the visible-light band can be used. Also, a filter incolor other than red, green, and blue can be used. Also, red, green andblue filters may have different colors. Accordingly, a degree of freedomin designing of a color filter becomes high and various kinds ofdesigning become possible.

Also, in the configuration of the imaging device of the presentinvention, it is preferable that the color filter includes four or morekinds of the filter parts corresponding to four or more kinds ofdifferent colors, one kind of the filter parts has a blockingcharacteristic in substantially the whole visible-light band and has atransmission characteristic in a fourth wavelength band on thelong-wavelength side of the visible-light band, and a transmissioncharacteristic of the optical filter and a transmission characteristicof each of the filter parts of the color filter are set in such a mannerthat the second wavelength band of the optical filter is included in thethird wavelength band and the fourth wavelength band.

According to such a configuration, in a case of eliminating an influenceof infrared light that passes through the second wavelength band byimage processing, it becomes possible to calculate signals, which aresubtracted from red, green and blue image signals, by using an infraredimage signal that is based on light passing through the infrared filterpart and it becomes possible to improve color reproducibility bycontrolling an influence of the infrared light that passes through thesecond wavelength band of the optical filter. That is, when it isassumed that the same quantity of infrared light in the same wavelengthband enters each pixel, quantity of light that passes through the secondwavelength band and passes through the red, green, blue, and infraredfilter parts does not vary greatly depending on a filter part andbecomes substantially the same. Thus, in this case, it is possible toremove light quantity based on the infrared light by subtractingquantity of the light that passes through the infrared filter part fromquantity of the light that passes through the red, green and blue filterparts.

Also, in the configuration of the imaging device of the presentinvention, it is assumed that infrared illumination is used in imagingof an infrared image, and setting is preferably performed in such amanner that a fifth wavelength band, which is a wavelength band ofinfrared light emitted from the infrared illumination, is included inthe third wavelength band and the fourth wavelength band and that thesecond wavelength band of the optical filter substantially overlaps withthe fifth wavelength band.

According to such a configuration, it is possible to make a wavelengthband of infrared light of the infrared illumination and a wavelengthband of the optical filter substantially the same. Also, the infraredlight of the infrared illumination can be efficiently used mainly in acase where photographing of an infrared image is performed at night byutilization of the infrared illumination. Also, in a case where awavelength band of infrared light of the infrared illumination isnarrow, it is possible to narrow down the second wavelength band of theoptical filter in accordance therewith. In this case, it is possible toreduce an influence of infrared light, which passes through the secondwavelength band, with respect to a visible image.

Advantageous Effects of Invention

According to an imaging sensor and an imaging device of the presentinvention, it is possible to control an influence of infrared light thatpasses through a second wavelength band in a visible light image byimage processing in a case of making it possible to performphotographing of a visible light image and an infrared image with anoptical filter having a transmission characteristic in a visible-lightband and a second wavelength band on an infrared side thereof.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view illustrating a first embodiment of thepresent invention and illustrating an imaging sensor.

FIG. 2 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and a color filter of the imaging sensor.

FIG. 3 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and the color filter of the imaging sensor and anemission spectrum of infrared illumination.

FIG. 4 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and the color filter of the imaging sensor and anemission spectrum of the infrared illumination.

FIG. 5(a) to FIG. 5(d) are views illustrating the same and fordescribing an array in the color filter of the imaging sensor, FIG. 5(a)being a view illustrating a conventional array with no infrared filterpart, and FIG. 5(b), FIG. 5(c), and FIG. 5(d) being views illustratingarrays with an infrared filter part.

FIG. 6 is a schematic view illustrating the same and illustrating animaging device including the imaging sensor.

FIG. 7 is a block diagram illustrating the same and for describingsignal processing in a signal processing unit of the imaging device.

FIG. 8 is a view illustrating the same and for describing internalprocessing in signal processing by the imaging device.

FIG. 9 is a schematic view illustrating an imaging device of a secondembodiment of the present invention.

FIG. 10 is a view for describing an array in a color filter of animaging sensor of a third embodiment of the present invention.

FIG. 11 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and the color filter of the imaging sensor.

FIG. 12 is a view illustrating the same and for describing an array inthe color filter of the imaging sensor.

FIG. 13 is a view illustrating the same and for describing an array inthe color filter.

FIG. 14 is a view illustrating the same and for describing an array inthe color filter.

FIG. 15 is a chart illustrating the same and for describing acharacteristic of each color filter.

FIG. 16 is a block diagram illustrating the same and for describingsignal processing in the imaging device.

FIG. 17 is a view for describing an array in a color filter of a fourthembodiment of the present invention.

FIG. 18 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and a color filter of blue B of the imaging sensor.

FIG. 19 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and a color filter of green G of the imaging sensor.

FIG. 20 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and a color filter of red R of the imaging sensor.

FIG. 21 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and a color filter of clear C of the imaging sensor.

FIG. 22 is a block diagram illustrating an imaging sensor of a fifthembodiment of the present invention.

FIG. 23 is a sectional view illustrating the same and illustrating theimaging sensor.

FIG. 24 is a sectional view illustrating the same and illustrating adifferent example of the imaging sensor.

FIG. 25 is a view illustrating the same and for describing an output ofimage signals in two systems by the imaging sensor.

FIG. 26 is a block diagram illustrating an imaging device of a sixthembodiment of the present invention.

FIG. 27 is a schematic view illustrating the same and illustrating animaging sensor.

FIG. 28 is a view illustrating the same and for describing an arraypattern in a color filter having an infrared region.

FIG. 29 is a graph illustrating the same and illustrating transmittancespectrums of DBPF and the color filter of the imaging sensor.

FIG. 30 is a graph illustrating the same and of an R signal fordescribing a reason why luminance is decreased in a highlight part.

FIG. 31 is a graph illustrating the same and of a G signal fordescribing a reason why luminance is decreased in a highlight part.

FIG. 32 is a graph illustrating the same and of a B signal fordescribing a reason why luminance is decreased in a highlight part.

FIG. 33 is a graph illustrating the same and of an IR signal fordescribing a reason why luminance is decreased in a highlight part.

FIG. 34 is a graph illustrating the same and illustrating a clip level,of the IR signal, corresponding to each of the color signals of RGB.

FIG. 35(a) to FIG. 35(c) are graphs illustrating the same and fordescribing subtraction of a clipped IR signal from each of the colorsignals of RGB, FIG. 35 (a) being a graph illustrating a case of the Rsignal, FIG. 35(b) being a graph illustrating a case of the G signal,and FIG. 35 (c) being a graph illustrating a case of the B signal.

FIG. 36(a) and FIG. 36(b) are views illustrating the same and fordescribing white-balance processing, FIG. 36(a) being a viewillustrating a region set to be white on a plane of an R−Y signal and aB−Y signal, and FIG. 36(b) being a view illustrating a range of aluminance level set to be white.

FIG. 37(a) to FIG. 37(c) are graphs illustrating the same and fordescribing a method of adjusting output levels of clipping RGB signalsafter subtraction of an IR signal.

FIG. 38 is a block diagram illustrating the same and illustrating asignal processing unit of an imaging processing device.

FIG. 39 is a block diagram illustrating the same and illustrating adifferent example of a signal processing unit.

FIG. 40 is a graph for describing a saturation level of an RGBC colorfilter of a seventh embodiment of the present invention.

FIG. 41 is a graph illustrating the same and for describing that asignal level of an IR signal acquired by calculation becomes higher thanthat of an actual IR signal.

FIG. 42 is a graph illustrating the same and of RGBC signals fordescribing a reason why luminance is decreased in a highlight part.

FIG. 43 is a block diagram illustrating the same and for describing aseparation device.

FIG. 44 is a block diagram illustrating the same and for describing aclip-level calculation device.

FIG. 45 is a graph illustrating the same and for describing clip levelsof the RGBC signals.

FIG. 46 is a graph illustrating the same and for describing a clip levelof the IR signal.

FIG. 47 is a graph illustrating the same and for describing clip levelsof the RGBC signals.

FIG. 48 is a block diagram illustrating the same and for describing aseparation device.

FIG. 49 is a graph illustrating the same and for describing clipprocessing of RGBCY signals.

FIG. 50 is a graph for describing a clip level of an IR signal of aneighth embodiment of the present invention.

FIG. 51 is a graph illustrating the same and for describing clipprocessing of RGBCY signals.

FIG. 52 is a block diagram illustrating the same and for describing aseparation device.

FIG. 53 is a block diagram illustrating the same and for describing anIR signal generation device.

DESCRIPTION OF EMBODIMENTS

In the following, a first embodiment of the present invention will bedescribed with reference to the drawings.

As illustrated in FIG. 1, an imaging sensor (image sensor) 1 of thepresent embodiment includes, for example, a sensor main body 2 that is acharge coupled device (CCD) image sensor, a color filter 3 in which eachof regions of red (R), green (G), blue (B), and infrared (IR) (filter ofeach color) is arranged in a predetermined array in a mannercorresponding to each pixel of the sensor main body 2, a cover glass 4that covers the sensor main body 2 and the color filter 3, and a doubleband pass filter (DBPF) 5 formed on the cover glass 4.

The sensor main body 2 is a CCD image sensor. A photodiode as alight-receiving element is arranged in each pixel. Note that the sensormain body 2 may be a complementary metal oxide semiconductor (CMOS)image sensor instead of the CCD image sensor.

The color filter 3 is provided in the sensor main body 2. In the colorfilter 3, an IR filter is added to a color filter 3 x including threefilter parts of R, G, and B which parts are arrayed in each pixel in ageneral Bayer array illustrated in FIG. 5(a). Note that in the filter inthe Bayer array, a basic pattern includes 16 filter parts in the colorsin four rows (horizontal sequence)×four columns (vertical sequence). Forexample, when each filter part is expressed with F, a row number, and acolumn number, F11, F12, F13, and F14 are in the first row, F21, F22,F23, and F24 are in the second row, F31, F32, F33, and F34 are in thethird row, and F41, F42, F43 and F44 are in the fourth row.

In the Bayer array, eight filter parts of F12, F14, F21, F23, F32, F34,F41, and F43 are G, four filter parts of F11, F13, F31, and F33 are R,and four filter parts of F22, F24, F42, and F44 are B. Note that thenumber of the filter parts in G is twice as large as the number offilter parts in each of R and B since a human eye has high sensitivitywith respect to green. Note that recognition with a human eye may not bepossible even when one with low sensitivity has high resolution.However, when one with high sensitivity has high resolution, possibilityof recognition with a human eye is high and an image with higherresolution is recognized. In this Bayer array, G filters are alternatelyarranged in a row direction (horizontal direction) and a columndirection (vertical direction) and a checkered pattern is formed. In theremaining part, the R filter parts and the B filter parts are arrangedin a manner not adjacent to each other.

On the other hand, as illustrated in FIG. 5 (b), by replacement of fourof the eight G filter parts in the Bayer array with IR, a color filter 3a including four R parts, four G parts, four B parts, and four IR partsis included as the color filter 3 of the present embodiment. That is, inthe basic array with four rows and four columns, four each of the fourkinds of R, G, B, and IR filter parts are arranged in such a manner thatthe same kinds of filter parts are arranged separately so as not to beadjacent to each other in the row direction and the column direction.One each of the R, G, B, and IR filter parts are arranged in each columnand two each of two kinds of filter parts among the R, G, B, and IRfilter parts are arranged in every other row.

More specifically, four filter parts of F11, F13, F32, and F34 are R,four filter parts of F12, F14, F31, and F33 are IR, four filter parts ofF21, F23, F42 and F44 are G, four filter parts of F32, F34, F41, and F42are B.

In this case, since the number of G filter parts is decreased, there isa possibility that resolution appears to be lower on a human eye.However, colors including IR are uniformly arranged and complementary(interpolation) processing becomes easy. Also, arrangement is made insuch a manner that a position of each color is shifted by one column inthe first and second rows and the third and fourth rows. In other words,in the first and second rows, and the third and fourth rows, thearrangement of the colors is horizontally reversed.

In such a manner, since each color is arranged by one in each column andeach color is arranged by two in every other row in the four-by-fourarrangement, resolution in a lateral direction (horizontal direction)becomes higher than that in a longitudinal direction (verticaldirection) and it is possible to control a decrease in resolution in thehorizontal direction due to provision of an IR filter part. Note thatthe color filter 3 a includes what is acquired by horizontal-reversal ofa pattern of the color filter 3 a illustrated in FIG. 5(b), what isacquired by vertical-reversal thereof, and what is acquired by180°-rotation thereof. Also, what is acquired by 90°-rotation or270°-rotation of that in a clockwise direction may be included. However,in what is acquired by 90°-rotation or 270°-rotation, resolution in avertical direction becomes higher than that in a horizontal direction.

Also, as illustrated in a color filter 3 b in FIG. 5(c), in the colorfilter 3, two of four B parts in a pattern of the color filter 3 x inthe Bayer array may be replaced with IR without reduction of G with highsensitivity of a human.

In this color filter 3 b, in the basic array with four rows and fourcolumns, eight G filter parts, four R filter parts, and two B filterparts and IR filter parts among the four kinds filter parts of R, G, B,and IR are arranged, the same kinds of filter parts being arrangedseparately in such a manner as not to be adjacent to each other in therow direction and the column direction.

More specifically, in the color filter 3 b, eight filter parts of F12,F14, F21, F23, F32, F34, F41, and F43 are G, and four filter parts ofF11, F13, F31, and F33 are R, two filter parts of F22 and F44 are B, andtwo filter parts of F24 and F42 are IR. In this case, resolution of Gand R is similar to that of the Bayer array although resolution of B islower than that of the Bayer array. Note that in the color filter 3 b,what is acquired by horizontal-reversal of a pattern of the color filter3 b illustrated in FIG. 5(c), what is acquired by vertical-reversalthereof, and what is acquired by 180°-rotation thereof are included.Also, what is acquired by 90°-rotation or 270°-rotation in the clockwisedirection has the same pattern with what is acquired byhorizontal-reversal or vertical-reversal.

Also, as illustrated in the color filter 3 c in FIG. 5(d), in order toimprove resolution of IR, two of four B parts in the pattern of thecolor filter 3 x in the Bayer array may be replaced with IR and two offour R parts may be replaced with IR without reduction of G. That is, inthe color filter 3 c, in the basic array with four rows and fourcolumns, eight G filter parts, four IR filter parts, and two R filterparts and B filter parts among the four kinds filter parts of R, G, B,and IR are arranged, the same kinds of filter parts being arrangedseparately in such a manner as not to be adjacent to each other in therow direction and the column direction.

More specifically, as illustrated in FIG. 5(d), in the color filter 3 c,eight filter parts of F12, F14, F21, F23, F32, F34, F41, and F43 are G,two filter parts of F11 and F33 are R, two filter parts of F22 and F44are B, and four filter parts of F13, F24, F31, and F42 are IR. In thiscase, resolution of G is kept and resolution of IR can be securedalthough resolution of R and B becomes lower than that of the Bayerarray. Note that in the color filter 3 c, what is acquired byhorizontal-reversal of a pattern of the color filter 3 b illustrated inFIG. 5(d), what is acquired by vertical-reversal thereof, and what isacquired by 180°-rotation thereof are included. Also, what is acquiredby 90°-rotation or 270°-rotation in the clockwise direction has the samepattern with what is acquired by horizontal-reversal orvertical-reversal.

As the R filter parts, the G filter parts, and the B filter parts in thecolor filter 3, known filters can be used. A transmittance spectrum ofeach of the R filter parts, the G filter parts, and the B filter partsin the present embodiment is in a manner illustrated in the graphs inFIG. 2, FIG. 3, and FIG. 4. In each of FIG. 2, FIG. 3, and FIG. 4, atransmittance spectrum of each of the red (R), green (G), blue (B), andinfrared (IR) filters in the color filter 3 is illustrated. A verticalaxis indicates transmittance and a horizontal axis is a wavelength. Arange of a wavelength in each graph includes the visible-light band anda part of the near-infrared band and indicates, for example, awavelength range of 300 nm to 1100 nm.

For example, as indicated by R (double line) in the graphs,transmittance of the R filter parts becomes substantially the maximum ata wavelength of 600 nm and a state in which the transmittance issubstantially the maximum is kept on a long-wavelength side thereof evenwhen 1000 nm is exceeded.

As indicated by G (wide-space broken line) in the graphs, the G filterparts have a peak at which transmittance becomes the maximum in a partwhere a wavelength is around 540 nm and have a part in which thetransmittance becomes the minimum in apart of around 620 nm on along-wavelength side thereof. Also, in the G filter parts, there is arising tendency on a long-wavelength side of the part where thetransmittance is the minimum, and the transmittance becomessubstantially the maximum at around 850 nm. On a long-wavelength side ofthat, a state in which the transmittance is substantially the maximum iskept even when 1000 nm is exceeded. As indicated by B (narrow-spacebroken line) in the graphs, the B filter parts have a peak at whichtransmittance becomes the maximum in a part where a wavelength is around460 nm and have a part in which the transmittance becomes the minimum ina part of around 630 nm on a long-wavelength side thereof. Also, thereis a rising tendency on a long-wavelength side of that and thetransmittance becomes substantially the maximum at around 860 nm. Astate in which the transmittance is substantially the maximum is kept ona long-wavelength side of that even when 1000 nm is exceeded. The IRfilter parts block light on a short-wavelength side from around 780 nmand light on a long-wavelength side from around 1020 nm. Transmittanceis substantially the maximum in a part from around 820 nm to 920 nm.

A transmittance spectrum of each of the R, G, B, and IR filter parts isnot limited to what is illustrated in FIG. 2 and the like. However, itis considered that a transmittance spectrum close to this is shown inthe color filter 3 that is currently used in general. Note that 1 on ahorizontal axis indicating transmittance does not mean that light istransmitted 100% but indicates, for example, the maximum transmittancein the color filter 3.

The cover glass 4 covers and protects the sensor main body 2 and thecolor filter 3.

Here, DBPF 5 is an optical filter formed on the cover glass 4. DBPF 5 isan optical filter that has a transmission characteristic in thevisible-light band, a blocking characteristic in a first wavelength bandadjacent to a long-wavelength side of the visible-light band, and atransmission characteristic in a second wavelength band that is a partin the first wavelength band.

That is, as indicated by DBPF (solid line) in the graph in FIG. 2, DBPF5 has high transmittance in two bands that are a visible-light bandindicated by DBPF (VR) and an infrared band (second wavelength band)that is at a slightly apart position on a long-wavelength side of thevisible-light band and is indicated by DBPF (IR). Also, DBPF (VR) as aband with high transmittance in the visible-light band is, for example,a wavelength band of around 370 nm to 700 nm. Also, DBPF (IR) as thesecond wavelength band with high transmittance on an infrared side is,for example, a band of around 830 nm to 970 nm.

In the present embodiment, a relationship between a transmittancespectrum of each filter part in the color filter 3 and a transmittancespectrum of DBPF 5 is prescribed as follows. That is, DBPF (IR) that isthe second wavelength band to transmit infrared light in thetransmittance spectrum of DBPF 5 is included in a wavelength band Awhich is illustrated in FIG. 2 and in which transmittance of all of theR filter parts, G filter parts, and B filter parts is substantially themaximum and the transmittance of the filter parts becomes substantiallythe same and in a wavelength band B to transmit light at substantiallythe maximum transmittance in the IR filter parts.

Here, the wavelength band A in which the transmittance of the R, G, andB filter parts becomes substantially the same is a part where adifference in the transmittance of the filter parts is 10% or smaller inthe transmittance.

Note that on a short-wavelength side of the wavelength band A, thetransmittance of the G and B filter parts becomes low while thetransmittance of the R filter parts becomes substantially the maximum.In DBPF 5, this part with a difference in the transmittance of the R, G,and B filter parts corresponds to a part, in which light is almostblocked and transmittance becomes the minimum in DBPF 5, between DBPF(VR) that is a part with high transmittance in the visible-light bandand DBPF (IR) that is a part with high transmittance in the secondwavelength band of the infrared light band. That is, on the infraredside, transmission of light in a part where a difference in thetransmittance of the R, G, and B filter parts becomes large is cut andlight is transmitted in the wavelength band A which is on along-wavelength side thereof and in which the transmittance of thefilter parts becomes substantially the maximum and the transmittancebecomes substantially the same.

From the above, in the present embodiment, since a region to transmitlight is not only in the visible-light band but also in the secondwavelength band on the infrared light side in DBPF 5 used instead of theinfrared light cut filter, there is an influence of light passingthrough the second wavelength band in color photographing with visiblelight. However, as described above, the second wavelength band does nottransmit light in a part where transmittance of the R, G, and B filterparts is different and only transmits light in a wavelength band inwhich transmittance of the filter parts becomes substantially themaximum and substantially the same.

Also, in a second wavelength low region in DBPF 5, light in a part wheretransmittance of the IR filter parts becomes substantially the maximumis transmitted. Thus, in a case where it is assumed that R, G, B, and IRfilter parts are respectively provided in extremely close four pixels towhich substantially the same light is emitted, in the second wavelengthband, light passes through the R filter part, the G filter part, the Bfilter part, and the IR filter part in substantially a similar manner.As light on an infrared side, light of substantially the same quantityreaches photodiodes of the pixels of the imaging sensor main body in thefilter parts including IR. That is, quantity of light, which passesthrough the second wavelength band on the infrared side, in lighttransmitted through each of the R, G, and B filters becomes similar toquantity of light passing through the IR filter part. In a case whereassumption is made in the above-described manner, basically, adifference between an output signal of a pixel assumed in theabove-described manner which signal is from the sensor main body 2 thatreceives light transmitted through each of the R, G, and B filters andan output signal of a pixel assumed in the above-described manner whichsignal is from the sensor main body 2 that receives light passingthrough the IR filter becomes an output signal in a visible light partof each of R, G, and B from which part infrared-side light passingthrough each of the R, G, and B filter parts is cut.

Actually, as illustrated in each pattern of the color filter 3 (3 a, 3b, and 3 c), any one of R, G, B, and IR filter parts is arranged in eachpixel of the sensor main body 2. Since it is likely that quantity oflight in each color which light is emitted to each pixel varies, forexample, it is possible to calculate luminance of each color in eachpixel by using a known interpolation method and to set a differencebetween this interpolated luminance of R, G, and B in each pixel andluminance of IR, which luminance is also interpolated, as luminance ofeach of R, G, and B in each pixel. Note that an image processing methodof removing an infrared light component from luminance of each of thecolors of R, G, and B is not limited to this. Any kind of method can beused as long as the method can finally cut an influence of light, whichpasses through the second wavelength band, from luminance of each of R,G, and B. In any method, since DBPF 5 cuts a part where transmittance ofthe R, G, and B filter parts is different for more than 10% on aninfrared side, that is, a part where transmittance is different for morethan a predetermined rate, processing of removing an influence ofinfrared light becomes easy in each pixel.

Also, this imaging sensor 1 is used as an imaging sensor in an imagingdevice that can perform both of color photographing and infrared lightphotographing. Generally, it is considered that normal photographing isperformed in the color photographing and the infrared photographing isperformed at night by utilization of illumination of infrared light,which cannot be recognized by a human, without utilization ofillumination of visible light. For example, it is considered that nightphotographing with infrared light by utilization of infrared-lightillumination is performed in various monitoring cameras and the like innight photographing at a place where night illumination is not necessaryor a place that is not preferably illuminated at night. Also,utilization in daytime photographing and night photographing forobservation of a wild animal, or the like is also possible.

In a case where infrared light photographing is used as nightphotographing, infrared-light illumination is necessary since quantityof infrared light is in short at night similarly to visible light. Atransmittance spectrum of DBPF 5 illustrated in FIG. 3 is determined inconsideration of a transmittance spectrum of each of the R, G, B, and IRfilter parts and an emission spectrum of light for infrared-lightillumination such as infrared-light LED for illumination. In FIG. 3, anemission spectrum IR-light of LED illumination is illustrated inaddition to transmittance spectrums R, G, B, and IR of the filter partsin the colors and a transmittance spectrum DBPF of DBPF 5 whichspectrums are similar to those in FIG. 2.

Similarly to DBPF illustrated in FIG. 2, a second wavelength band thatis indicated by DBPF (IR) and that is a part to transmit infrared lightin DBPF illustrated in FIG. 3 is included in a wavelength band A whichis illustrated in FIG. 2 and in which transmittance of all of the Rfilter part, G filter part, and B filter part becomes substantially themaximum and transmittance of the filter parts becomes substantially thesame and in a wavelength band B that transmits light at the maximumtransmittance of the IR filter part.

In addition, substantially a whole wavelength band which is included inboth of the above-described wavelength band A and wavelength band B andin which an emission spectrum of infrared-light illumination reaches apeak is included in a wavelength band of DBPF (IR). Note that in a casewhere infrared light photographing is performed not with natural lightat night but with infrared-light illumination, it is not necessary thatthe second wavelength band indicated by DBPF (IR) is wider than a peakwidth of an optical spectrum of the infrared-light illumination. In acase where a spectrum of the infrared-light illumination is included inboth of the above-described wavelength band A and wavelength band B, apeak part of transmittance of DBPF 5 indicated by DBPF (IR) may beprovided as the second wavelength band in a peak width substantiallysimilar to a peak width of a peak of the emission spectrum of theinfrared-light illumination with around 860 being the peak, for example.

That is, in FIG. 3, the peak in the emission spectrum of theinfrared-light illumination which spectrum is indicated by IR-light ison a short-wavelength side of the above-described wavelength band A andwavelength band B. Also, the second wavelength band of DBPF which partis indicated by DBPF (IR) substantially overlaps with the peak of theemission spectrum in IR-light in a part on the short-wavelength side ofthe wavelength band A and the wavelength band B.

Also, similarly to a graph illustrated in FIG. 3, in the graphillustrated in FIG. 4, an emission spectrum of infrared-lightillumination is added to the graph illustrated in FIG. 2 and a secondwavelength band which is indicated by DBPF (IR) and which is a partwhere transmittance on an infrared side of a transmittance spectrum ofDBPF 5 is high is adjusted to a peak of the above-described emissionspectrum, which is indicated by IR-light, of the infrared-lightillumination.

In FIG. 4, as infrared-light illumination, one with a wavelength of apeak of an emission spectrum being longer than that in the caseillustrated in FIG. 3 is used. This peak is included in theabove-described wavelength band A and wavelength band B and is on along-wavelength side of the wavelength band A and the wavelength band B.In accordance with this, a second wavelength band indicated by DBPF (IR)in DBPF 5 is provided in such a manner as to substantially overlap witha peak, which is indicated by IR-light, of the infrared illumination inthe above-described wavelength band A and wavelength band B.

Note that the second wavelength band of DBPF 5 may be what isillustrated in any of FIG. 2, FIG. 3, and FIG. 4. The second wavelengthband only needs to be included in both of the above-described wavelengthband A and wavelength band B. Also, in a case where a wavelength band inwhich an emission spectrum of infrared-light illumination used in nightinfrared light photographing at night reaches a peak is determined, itis preferable that the wavelength band is included in both of theabove-described wavelength band A and wavelength band B and the secondwavelength band of DBPF 5 is adjusted to the peak of the emissionspectrum of the infrared-light illumination.

In such an imaging sensor, the second wavelength band that transmitslight on an infrared side of DBPF 5 is included, on an infrared side ofeach of R, G, B, and IR filter parts, in the wavelength band A in whichtransmittance of each filter part becomes substantially the maximum andtransmittance of the filter parts becomes substantially the same and thewavelength band B in which transmittance of an IR filter part becomessubstantially the maximum. In other words, on a long-wavelength side ofa visible-light band, transmittance of only an R fill part among R, G,and B filters becomes substantially the maximum and transmittance of theG and B filter parts are not substantially the maximum. Thus, light in apart where the transmittance of the R, G, and B filter parts is notsubstantially the same and is different is cut by DBPF 5.

That is, in each of the R, G, B, and IR filter parts, light in thesecond wavelength band is transmitted on the infrared side. Thus,transmittance on the infrared side in the filter parts is substantiallythe same. When the same quantity of light to be in the second wavelengthband is emitted, transmission light quantity in the R, G, B, and IRfilter parts becomes the same. Accordingly, as described above, it ispossible to correct a color based on an output signal from a pixelcorresponding to each of the R, G, and B filter parts and to easilyacquire an image in which an influence due to infrared light passingthrough a second wavelength band of a color in color photographing iscontrolled.

Also, since the second wavelength band is made to correspond to a peakof the emission spectrum of the infrared-light illumination whichspectrum is included in the above-described wavelength band A andwavelength band B, light of the infrared-light illumination can beefficiently used. Also, it is possible to narrow down a width of thesecond wavelength band and to reduce an influence of infrared lightpassing through the second wavelength band in color photographing.

FIG. 6 is a view illustrating an imaging device 10 using the imagingsensor 1 of the present embodiment. The imaging device 10 includes alens for imaging 11, an imaging sensor 1 with DBPF 5, and a signalprocessing unit (signal processing device) 12 that processes an outputsignal 13 output from the imaging sensor 1 and that performs theinternal processing, the image processing of removing an influence ofinfrared light passing through the second wavelength band in colorphotographing, or image processing such as gamma correction, whitebalancing, or RGB matrix correction with respect to an image signal.From an image processing unit, an output signal 14 of a visible colorimage and an output signal 15 of an infrared light image can be output.

The lens 11 is included in an optical system that forms an image on theimaging sensor 1 of the imaging device 10. For example, the lens 11includes a plurality of lenses.

FIG. 7 is a block diagram illustrating signal processing in the signalprocessing unit 12 of the imaging device 10. Output signals of pixels ofR, G, B, and IR are respectively transmitted internal processing blocks21 r, 21 g, 21 b, and 21 ir. For example, as illustrated in FIG. 8, in acase of the above-described color filter 3 b, in the internal processingblocks 21 r, 21 g, 21 b, and 21 ir, signals of R, G, B, and IR arerespectively converted by interpolation processing into image data 20 rin which all pixels are expressed in red R, image data 20 g in which allpixels are expressed in green G, image data 20 b in which all of thepixels are expressed in blue B, and image data 20 ir in which all pixelsare expressed in infrared IR in image data in each frame. Note that asan interpolation processing method, a known method can be used.

Then, in infrared light removal signal creating blocks 22 r, 22 g, 22 b,and 22 ir, a signal that is subtracted from each R, G, and B signals isgenerated from an IR signal in order to remove an influence of infraredlight received from of the above-described second wavelength band. Thesesignals respectively created for R, G, and B by the infrared lightremoval signal creating blocks 22 r, 22 g, and 22 b are respectivelysubtracted from the R, G, and B signals. In this case, as describedabove, an IR signal is basically removed from each of R, G, and Bsignals in the same pixel. Thus, the processing becomes easy. Actually,since sensitivity is different in each pixel of each color due to acharacteristic or the like of a filter part in each pixel, signals to besubtracted from the R, G, and B signals are created for R, G, and Bimages from an IR signal.

Then, with respect to each of the R, G and B signals, known RGB matrixprocessing of converting each of the R, G and B signals and correcting acolor by using a determinant, known white-balance processing of makingoutput values of the R, G, and B signals the same in a part to be whitein an image, and known gamma correction that is correction for an imageoutput to a display or the like are performed in an image processingblock 23. Subsequently, in a luminance matrix block 24, a signal withluminance Y is generated by multiplication of the R, G, and B signals bya coefficient. Also, color difference signals R−Y and B−Y are calculatedby division of the signal of blue B and the signal of red R by thesignal with the luminance Y and the signals of Y, R−Y, and B−Y areoutput.

Also, an IR signal is basically output as a gradation image in black andwhite.

As described with reference to the imaging sensor 1, in such an imagingdevice 10, image processing of removing an influence of infrared lightfrom a color image can be easily performed and a visible color imagewith high color reproducibility can be acquired. Also, it is possible toreduce a development cost of such an imaging device.

As described above, in a case where an influence of infrared lightpassing through the second wavelength band is cut by image processing invisible light photographing, a part in which transmittance of the R, G,and B filter parts is greatly different from each other is physicallycut by DBPF. In the image processing, processing of cutting IR light ina part where transmittance becomes substantially the maximum on aninfrared side of each of the R, G, and B filter parts is performed. Inthis case, the image processing becomes easy and it becomes possible toacquire color image data having color reproducibility similar to that ina case where a conventional infrared light cut filter is used.

For example, light that passes through each of the R, G, and B filterparts and reaches a photodiode is visible light transmitted through eachfilter part in a visible light region and infrared light that passesthrough the second wavelength band, that becomes similar in the R, G,and B filter parts, and that becomes similar in the IR filter part.Thus, for example, by subtracting an IR output signal of after theinterpolation processing, which signal is corrected in accordance with acharacteristic such as sensitivity based on a filter part in each color,from each of R, G, and B output signals of after the interpolationprocessing in the imaging sensor 1, it is possible to acquire colorreproducibility close to that in a case where the infrared cut filter isused.

Note that when a wavelength band in which there is a difference largerthan 10% in transmittance of the R, G and B filter parts is included inthe second wavelength band that transmits light, it becomes practicallydifficult to calculate quantity of infrared light to be cut from thelight that passes through each filter part and it is difficult toacquire image data, which has color reproducibility similar to that in acase where the infrared light cut filter is used, by image processing.

Next, an imaging device according to a second embodiment of the presentinvention will be described.

As illustrated in FIG. 9, in an imaging device 10 a of the secondembodiment, DBPF 5 is not provided on an imaging sensor 1 but DPBF isprovided on a lens 11.

The imaging device 10 a includes a lens for imaging 11 with DBPF 5, animaging sensor 1, and a signal processing unit 12 that processes anoutput signal 13 output from the imaging sensor 1 and that performs theinternal processing, the image processing of removing an influence ofinfrared light passing through a second wavelength band in colorphotographing, or image processing such as gamma correction, whitebalancing, or RGB matrix correction with respect to an image signal.From an image processing unit, an output signal 14 of a visible colorimage and an output signal 15 of an infrared light image can be output.

DBPF 5 and a color filter 3 are similar to DBPF 5 and the color filter 3of the first embodiment. A relationship between transmittance of each ofR, G, B, and IR filter parts of the color filter 3 and a secondwavelength band DBPF (IR) of DPBF 5 is also similar to that in the firstembodiment. Thus, it is possible to acquire an effect similar to that ofthe imaging device 10 of the first embodiment even when DBPF 5 isprovided in the lens 11 unlike the first embodiment. Note that DBPF 5may be provided anywhere as long as provision is in an optical system ofthe imaging device 10 a, light in a visible-light band (DBPF (VR)) andin a second wavelength band on an infrared side (DBPF (IR)) istransmitted, and light on a short-wavelength side of the visible-lightband, on a long-wavelength side of the second wavelength band, and in awavelength band between the visible-light band and the second wavelengthband is blocked with respect to light that reaches the imaging sensor 1.

Next, an imaging sensor and an imaging device of a third embodiment ofthe present invention will be described. In an imaging sensor 1 and animaging device of the third embodiment, a part of a configuration of acolor filter 3 and a method of removing an IR component from each of RGBsignals are different but the other configuration is similar to that ofthe first embodiment. The color filter 3 and the method of removing anIR component will be described in the following.

For example, as illustrated in FIG. 10, in a color filter 3 e (firstconfiguration of RGBC) in the present embodiment, two of four B partsare C, two of four R parts are C, and four of eight G parts are C in apattern of a color filter 3 x in the above-described Bayer array. Thatis, in the color filter 3 e, in a basic array of four rows and fourcolumns, four G filter parts, eight C filter parts, and two R filterparts and B filter parts among filter parts of four kinds of R, G, B,and C are arranged, the same kinds of filter parts being arrangedseparately in such a manner as not to be adjacent to each other in a rowdirection and a column direction. Thus, the eight C filter parts arearranged in a checkered pattern. Here, C indicates a transparent stateas a clear filter part and basically has a transmission characteristicfrom a visible-light band to a near-infrared wavelength band. Here,C=R+G+B in the visible-light band. Note that since three colors of RGBare transmitted, C indicating clear can be called white light, that is,white (W) and C=W=R+G+B is acquired. Thus, C corresponds to lightquantity in substantially a whole wavelength band of the visible-lightband.

Here, as illustrated in a graph illustrating a transmittance spectrum(spectral transmission characteristic) of the color filter 3 e and DBPF5 in FIG. 11, there is a transmission characteristic on along-wavelength side of a visible-light band in each of the R, G, and Bfilter parts and light is also transmitted on the long-wavelength sideof the visible-light band in the C filter part that is a clear filterpart. Similarly to the first embodiment, by utilization of DBPF 5 withrespect to this, infrared that passes through the long-wavelength sideof the visible-light band is limited to be a second wavelength band.Thus, quantity of light passing through the R, G, B, and C filter partsand DBPF 5 becomes substantially the same (approximate) in each of theR, G, B, and C filter parts and transmission characteristicscorresponding to wavelengths of the R, G, B, and C filter parts aredifferent in the visible-light band.

Note that in each of the first and second embodiments, infrared thatpasses through the long-wavelength side of the visible-light band isalso limited to be a second wavelength band. Thus, quantity of lightpassing through the R, G, B, and IR filter parts and DBPF 5 becomessubstantially the same in each of the R, G, B, and IR filter parts andtransmission characteristics corresponding to wavelengths of the R, G,B, and IR filter parts are different in the visible-light band.

Accordingly, in the third embodiment, it is also possible to perform IRcorrection of each pixel accurately and to generate a visible image withhigh color reproducibility. That is, even when the above-described IRfilter part, which has a blocking characteristic in substantially awhole wavelength region of a visible-light band and has a transmissioncharacteristic in infrared on a long-wavelength side of thevisible-light band, is not included unlike the first embodiment, an IRsignal can be calculated by the following expression since the C filterpart is included.

In the following description, C (W), R, G, B, and IR indicate levels ofan output signal from the imaging sensor 1. Here, it is assumed that C(W), R, G, and B indicate levels in the visible-light band and do notinclude an infrared component.

Here, when the color filter 3 e is designed in a manner of C=W≈R+G+B andan IR signal to be removed from each of RGB signals is IR′,IR′=((R+IR)+(G+IR)+(B+IR)−(C+IR))/2=IR+(R+G+B−C)/2

IR′≈IR is acquired. Note that IR indicates an actual value acquired bymeasurement or the like and IR′ indicates a value acquired bycalculation. It is possible to perform IR correction by subtracting IR′from each filter.

That is,

R filter (R+IR):R′=(R+IR)−IR′=R−(R+G+B−C)/2

G filter (G+IR):G′=(G+IR)−IR′=G−(R+G+B−C)/2

B filter (B+IR):B′=(B+IR)−IR′=B−(R+G+B−C)/2

C (=W) filter (W+IR):W′=(C+IR)−IR′=C−(R+G+B−C)/2.

Accordingly, even when a clear C filter part is used instead of an IRfilter part in the color filter 3, it is possible to approximatetransmittance of IR in each filter part by DBPF 5 and to improve colorreproducibility by calculating an IR component and removing this from asignal of each filter part as described above.

Note that in such a calculation, for example, R+IR, G+IR, B+IR, and C+IRare calculated in each pixel by an interpolation method as describedabove and the above calculation is performed in each pixel. Note thatdesigning is performed in the manner of C=W−R+G+B. However, accuratematching with this expression is not necessary. There may be a deviationdue to an error or the like and there may be, for example, a deviationof around 10% as long as there is approximation.

Also, since C becomes R+G+B and is easily saturated due to largequantity of light received in a pixel, quantity of received light in thevisible-light band may be decreased, quantity of received light may bedecreased in a wavelength band including an infrared wavelength band andthe visible-light band in the C filter part or an amount of electriccharge accumulated with respect to the quantity of received light may bedecreased in an element part included in each pixel. In that case, it isnecessary to change the above-described expression accordingly.

Note that FIG. 12 is a view illustrating a different array of R, G, B,and C color filters, one each of R, G, B, and C parts being uniformlyarranged in a two-by-two array.

Also, in a case of a conventional Bayer array of R, G, and B, one eachof R and B parts, and two G parts are arranged in a two-by-two array asillustrated in FIG. 13.

Also, a two-by-two array of R, G, B, and IR color filters, in whicharray one G part in the conventional array that does not include C or IRis replaced with IR, becomes an array in which one each of R, G, B, andIR parts are arranged as illustrated in FIG. 14.

As such a color filter, there is a difference in characteristics asillustrated in FIG. 15 in the first configuration of RGB-C illustratedin FIG. 10, a second configuration of RGB-C illustrated in FIG. 12, aconventional RGB array illustrated in FIG. 13 (Bayer array), and anexample of an RGB-IR array illustrated in FIG. 14. Note that C does notinclude color information such as RGB but includes information ofluminance as light quantity.

Thus, due to a checkered arrangement of C, luminance resolution is highbut there are sparse RGB pixels in an RGB-C (first configuration)sensor.

In addition, since an arrangement becomes asymmetrical, resolution islow and moire is likely to be generated. However, since resolutionrequired to a color signal is ½ or lower of that of a luminance signaland is low, there is no problem. Also, sensitivity is high.

RGB-C (second configuration) has luminance resolution and colorresolution equivalent to those of a conventional RGB sensor andsensitivity higher than that of the RGB sensor. An RGB-IR sensor haslower sensitivity and lower luminance resolution than the RGB sensorsince IR that does not have a transmission characteristic in thevisible-light band is provided.

That is, it is likely that a color filter having C has an advantage inresolution or sensitivity compared to a color filter having IR in eachof the above-described first embodiment and second embodiment.

FIG. 16 is a block diagram illustrating signal processing in the signalprocessing unit 12 in FIG. 9. As the imaging sensor 1, an RGB-C sensor 1including the above-described RGB-C color filter 3 e is included. Also,a lens 11 included in an optical system and DBPF 5 are included.

Signals of R+IR, G+IR, B+IR, and C+IR are input from the RGB-C sensor 1into a separation device 51 to perform color separation, IR separation,and IR correction. By interpolation processing, the IR correction, andthe like, each of R, G, B, W, and IR signals is calculated and output ineach pixel. This processing is performed on the basis of calculationusing the above expression.

Each of the R, G, and B signals among the R, G, B, W, and IR signalsoutput from the separation device 51 is transmitted to a color matrixdevice 52. Then, known RGB matrix correction or the like is performedand RGB signals are output. Also, the R, G, B, W, and IR signals fromthe separation device 51 are transmitted to a luminance generationdevice 53. A luminance signal is generated from each signal on the basisof an expression of calculating set luminance.

The RGB signals output from the color matrix device are input into agamma processing and color difference generation device 54 and knowngamma processing is performed. Also, for example, B−Y and R−Y signalsare generated as color difference signals. Also, after noise reductionas a signal in a predetermined wavelength band in a noise reductiondevice 56 through a band pass filter (BPF) 55, a signal output from eachof the separation device 51 and the RGB-C sensor 1 is amplified in anenhancement processing device 57 along with a luminance signal outputfrom the luminance generation device 53 and is output as a luminancesignal (Y signal) of a luminance/color difference signal after gammaprocessing in the gamma processing device 58.

Also, an IR signal output from the separation device 51 is output as anIR signal through an enhancement processing device 59 and a gammaprocessing device 60. Note that in the processing of an image signal,clip processing (described later) is performed and the clip processingwill be described later.

Next, an imaging sensor and an imaging device of a fourth embodiment ofthe present invention will be described. In the fourth embodiment, eachcolor of a color filter is generalized and it is indicated that a colorfilter of the present invention is not limited to RGB-IR or RGB-C. Inthe following, a method of removing an IR component in an imaging sensorincluding a color filter with generalized four color filter parts willbe described. Note that the four colors (four kinds) of filter partsbasically have different transmission characteristics corresponding towavelengths in a visible-light band. In addition, a third wavelengthband, in which a difference in transmittance with a difference filterpart on a long-wavelength side of the visible-light band becomes 10% orsmaller, is included in a wavelength band including the above-describedsecond wavelength band of DBPF and the second wavelength band of DBPF 5is included in this third wavelength band. Accordingly, in a case wherea color filter and DPBF 5 are used, transmission characteristicscorresponding to wavelengths on an infrared side of the visible-lightband are approximate to each other in the filter parts in the colors.

Moreover, in a filter arrangement of four kinds of pixels, IR can beseparated when a color filter is designed in the following condition.

In the filter arrangement, it is preferable that one each of four kindsof filter parts A, B, C, and D are included in a two-by-two arrangementas illustrated in FIG. 17.

Also, it is preferable that the filter parts A, B, C, and D are designedin such a manner that the following relationship is established in avisible wavelength band when possible.

That is, in the visible-light band,

it is assumed that KaA+KbB+KcC+KdD≈0.

Note that each of A, B, C, and D indicates a level of an output signalfrom the imaging sensor 1 in a visible-light band of each filter part.

It is assumed that transmittance of IR becomes substantially constant inthe above-described third wavelength band of each of the A, B, C, and Dfilter parts in an IR region on a long-wavelength side of thevisible-light band. Note that transmittance of IR may be substantiallyan integer multiple of certain IR transmittance in each of the A, B, C,and D filter parts. When designing is performed in such a manner (here,it is assumed that IR transmittance is constant as described above),Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR)≈IR(Ka+Kb+Kc+Kd).

Thus, an IR signal can be calculated byIR′=(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd).

By the following calculation, it is possible to correct an IR componentincluded in each of A, B, C, and D pixels.A′=(A+IR)−(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)=A−(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd)B′=B+IR−(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)=B−(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd)C′=C+IR−(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)=C−(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd)D′=D+IR−(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)=D−(KaA+KbB+KcC+KdD)/(Ka+Kb+KC+Kd)

Here, an error amount is

(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd). This error amount can be corrected inan RGB matrix.

Actually, since transmittance of an IR component in the filter partsvaries slightly, correction is performed with a coefficient-correctedsignal as described in the following.A′=A+IR*KIRa−KIRa(Ka(A+IR*KIRa)+Kb(B+IR*KIRb)+Kc(C+IR*KIRc)+Kd(D+IR*KIRd))/(Ka*KIRa+Kb*KIRb+Kc*KIRc+Kd*KIRd)B′=B+IR*KIRb−KIRb(Ka(A+IR*KIRa)+Kb(B+IR*KIRb)+Kc(C+IR*KIRc)+Kd(D+IR*KIRd))/(Ka*KIRa+Kb*KIRb+Kc*KIRc+Kd*KIRd)C′=C+IR*KIRc−KIRc(Ka(A+IR*KIRa)+Kb(B+IR*KIRb)+Kc(C+IR*KIRc)+Kd(D+IR*KIRd))/(Ka*KIRa+Kb*KIRb+Kc*KIRc+Kd*KIRd)D′=D+IR*KIRd−KIRd(Ka(A+IR*KIRa)+Kb(B+IR*KIRb)+Kc(C+IR*KIRc)+Kd(D+IR*KIRd))/(Ka*KIRa+Kb*KIRb+Kc*KIRc+Kd*KIRd)

Note that a spectral transmission characteristic of each filter of whenDBPF is used is in a manner illustrated in FIG. 11. Note that an exampleof filter parts is an example of using four kinds of filter parts ofR+IR, G+IR, B+IR, and C+IR. However, the filter parts are not limited toR+IR, G+IR, B+IR, and C+IR as long as a color filter is designed in sucha manner that IR parts are constant or in a relationship of an integermultiple to each other and KaA+KbB+KcC+KdD≈0 is acquired.

Spectral transmission of a combination of the B filter part and DBPF 5is illustrated in FIG. 18, spectral transmission of a combination of theG filter part and DBPF 5 is illustrated in FIG. 19, spectraltransmission of a combination of the B filter part and DBPF 5 isillustrated in FIG. 20, and spectral transmission of a combination ofthe C (W) filter part and DBPF 5 is illustrated in FIG. 21.

As expressed in each of the above-described expressions, each spectraltransmission characteristic is the sum of four kinds of transmittance ofa visible R transmission region, a visible G transmission region, avisible B transmission region, and an IR transparent region. From theabove, it is possible to calculate a signal value of each of a visible Rtransmission region, a visible G transmission region, a visible Btransmission region, and an IR transmission region from values of fouror more kinds of filters. Note that a spectral transmissioncharacteristic is determined on the basis of an expression expressingeach of spectral transmission characteristics of A, B, C, and D. Sixkinds are determined from combinations of spectral transmissioncharacteristics of two of these filters.

Next, a fifth embodiment of the present invention will be described.

In a case where an imaging sensor 1 that outputs a signal in avisible-light band and an infrared signal on a long-wavelength sidethereof is mounted in a smartphone or the like and in a case where acircuit that processes a signal from the imaging sensor 1 is on a systemon chip (SOC) that functions as a main arithmetic processing device onthe smartphone side, when RGB signals output from the imaging sensorhave an IR component as described above and processing of removing thisIR component is necessary, it may be necessary to change a design ofSOC. Also, it becomes necessary to provide a circuit for performing theabove-described IR correction in the smartphone in addition to SOC. Inthese cases, in introduction of an imaging sensor, which can output bothof visible-light and infrared image signals, into a device such as asmartphone, a cost is increased in a part other than the imaging sensor.

Thus, in the present embodiment, a circuit in which an image signaloutput in a visible-light band from the imaging sensor 1 is a signalsimilar to a conventional one is embedded in the imaging sensor. Asillustrated in FIG. 22, an imaging sensor 101 includes an imaging unit102 that has a configuration substantially similar to that of theabove-described imaging sensor 1, and an IR correction/separationcircuit 103. For example, from the IR correction/separation circuit 103,a visible RGB signal and an IR signal are output. Note that the IRcorrection/separation circuit 103 includes a circuit that performsprocessing related to clipping of a signal level (described later).

In this case, the number of pins of the imaging sensor 101 is increasedto output an RGB signal and an IR signal. However, it is possible tocontrol the increase in the number of pins by an output in a serialoutput standard such as CSI-2. Here, a visible RGB signal is output asan RAW signal output of an imaging sensor in a Bayer array of RGBG or asa YUYV (YCb or YCr) signal. Also, when an IR signal is output as amonochromatic signal, it is possible to utilize visible andnear-infrared imaging simultaneously by an RGB-IR sensor or an RGB-Csensor without changing SOC of a smartphone or a feature phone.

For example, as illustrated in FIG. 23, a structure of the imagingsensor 101 becomes a multi-layer stack structure. For example, a chip ofan integrated circuit included in the IR correction/separation circuit103 is mounted on one substrate 110 and a chip included in the imagingunit 102 is arranged thereon. A cover glass 111 is arranged on theimaging unit 102. A solder ball 115 is arranged on a bottom surface ofthe substrate 110.

Also, as illustrated in FIG. 24, in the structure of the imaging sensor101, two substrates 110 may be arranged in a vertically overlappedmanner with a space therebetween, a chip of the IR correction/separationcircuit 103 may be arranged on the lower substrate 110, a chip of theimaging unit 102 may be arranged on the upper substrate 110, and thecover glass 111 may be arranged thereon.

Currently, it is possible to realize a small package with a multi-layerstack structure. That is, it is possible to vertically stack the IRcorrection/separation circuit 103 and the imaging unit 102 as describedabove and to form a sensor in one package. It is possible to realize theimaging sensor 101, into which an IR correction/separation circuit isembedded, in one package even by this method. In such a manner, it ispossible to realize a small sensor that can performvisible/near-infrared light imaging simultaneously and that can be usedin a smartphone or the like even when a correction/separation circuit isnot embedded. In a smartphone, a function of biometric authenticationsuch as iris authentication, or 3D capture is included by an IR sensorand imaging of a visible video/still image can be realized by onesensor.

That is, in a case where biometric authentication or the like isperformed with an infrared sensor, for example, it is considered thatthe infrared sensor is provided in a manner separated from a camera andthat an increase in a cost, deterioration in space efficiency, or thelike is caused in this case due to addition of a new infrared sensor.However, it is possible to control these.

FIG. 25 is a schematic view illustrating a signal output, which issimilar to that of a Bayer arrangement of RGB, by IRcorrection+interpolation. In horizontal/vertical scanning, R/IR and G/Bare output in a line-sequential manner of R, IR, R, IR . . . /G, B, G, B. . . . With respect to this signal, IR correction/IR separation isperformed. Also, at a position of an IR pixel, a G′ signal is generatedby interpolation from a G signal in the vicinity. Visible R/G′ and G/Bsignals and a separated IR signal are output in a line-sequential mannerof R, G′, R, G′ . . . /G, B, G, B . . . . Since visible R/G′ and G/Bsignals are in an output format that is the same with that of aconventional sensor in the Bayer array of RGB, processing can beperformed in a signal processing circuit that is the same with aconventional one on a signal processing side. Also, since an IR signalis still a monochromatic signal, it is also possible for a signalprocessing unit to perform processing by normal monochrome signalprocessing.

By the above configuration, it is possible to make a change in aprocessing circuit as small as possible and to process an output of anRGB-IR sensor in a conventional signal processing circuit. Also, itbecomes possible to implement a new function such as biometricauthentication or 3D capture by IR imaging without an increase in aspace in a smartphone or the like.

Also, an RGB-IR sensor has been described above. However, insimultaneous visible/near-infrared imaging using an RGB-C sensor, it isalso possible to acquire a similar effect by embedding a circuit for IRcorrection/separation into a sensor and by forming a configuration insuch a manner that a visible signal, which is in an output format of aconventional sensor in the Bayer array of RGB, and an IR signal areoutput.

Next, a sixth embodiment of the present invention will be described.

As illustrated in FIG. 26, an imaging device 10 of the presentembodiment includes a lens 11 that is an optical system forphotographing, an imaging sensor 1 with DBPF 5, and a signal processingunit (signal processing device: subtraction control device) 12 thatprocesses an output signal 13 output from the imaging sensor 1 and thatperforms the internal processing, the image processing of removing aninfluence of infrared light passing through a second wavelength band incolor photographing, or image processing such as gamma correction, whitebalancing, or RGB matrix correction with respect to an image signal. Anoutput signal 14 of a visible color image (visible image signal) and anoutput signal 15 of an infrared light image (infrared image signal) canbe output from the signal processing unit 12.

The lens 11 is included in an optical system that forms an image on theimaging sensor 1 of the imaging device 10. For example, the lens 11includes a plurality of lenses.

For example, as illustrated in FIG. 27, the imaging sensor (imagesensor) 1 includes a sensor main body 2 that is a charge coupled device(CCD) image sensor, a color filter 3 in which each of regions of red(R), green (G), blue (B), and infrared (IR) (filter of each color) isarranged in a predetermined array in a manner corresponding to eachpixel of the sensor main body 2, a cover glass 4 that covers the sensormain body 2 and the color filter 3, and a double band pass filter (DBPF)5 formed on the cover glass 4.

The sensor main body 2 is a CCD image sensor. A photodiode as alight-receiving element is arranged in each pixel. Note that the sensormain body 2 may be a complementary metal oxide semiconductor (CMOS)image sensor instead of the CCD image sensor.

The color filter 3 is provided in the sensor main body 2. Here, a colorfilter in a Bayer array with regions of red R, green G, and blue B andwithout a region of infrared IR has four-by-four 16 regions serving as abasic pattern. Here, eight regions are G regions, four regions are R,and four regions are B. On the other hand, as illustrated in FIG. 28, inthe color filter 3 of the present embodiment, four among the eight Gregions in the Bayer array are replaced with IR regions, whereby thereare four R regions, four G regions, four B regions, and four IR regions.Note that a color filter including an IR region is not limited to thecolor filter 3 illustrated in FIG. 28 and a color filter with variousarrays can be used. Also, each of RGB regions is in a general RGBfilter, has a peak of transmittance in a wavelength range of each color,and has transitivity in a near-infrared wavelength region. Thus, the redregion is R+IR, the green region is G+IR, and the blue region is B+IR inFIG. 28.

A transmittance spectrum of each of the R region, the G region, and theB region in the present embodiment is in a manner illustrated in a graphin FIG. 29. That is, a transmittance spectrum of each of the red (R),green (G), blue (B), and infrared (IR) filters in the color filter 3 isillustrated. A vertical axis indicates transmittance and a horizontalaxis indicates a wavelength. A range of a wavelength in the graphincludes a visible-light band and a part of a near-infrared band andindicates, for example, a wavelength range of 300 nm to 1100 nm.

For example, as indicated by R (double line) in the graph, transmittanceof the R region becomes substantially the maximum at a wavelength of 600nm and a state in which the transmittance is substantially the maximumis kept on a long-wavelength side thereof even when 1000 nm is exceeded.As indicated by G (wide-space broken line) in the graph, the G regionhas a peak at which transmittance becomes the maximum in a part, inwhich a wavelength is around 540 nm, and has a part at which thetransmittance becomes the minimum in a part of around 620 nm on along-wavelength side thereof. Also, in the G region, there is a risingtendency on a long-wavelength side of the part where the transmittanceis the minimum, and the transmittance becomes substantially the maximumat around 850 nm. On a long-wavelength side of that, a state in whichthe transmittance is substantially the maximum is kept even when 1000 nmis exceeded. As indicated by B (narrow-space broken line) in the graphs,the B region has a peak at which transmittance becomes the maximum in apart, in which a wavelength is around 460 nm, and has a part in whichthe transmittance becomes the minimum in a part of around 630 nm on along-wavelength side thereof. Also, there is a rising tendency on along-wavelength side of that and the transmittance becomes substantiallythe maximum at around 860 nm. A state in which the transmittance issubstantially the maximum is kept on a long-wavelength side of that evenwhen 1000 nm is exceeded. The IR region blocks light on ashort-wavelength side from around 780 nm and light on a long-wavelengthside from around 1020 nm. Transmittance is substantially the maximum ina part from 820 nm to 920 nm.

A transmittance spectrum of each of the R, G, B, and IR regions is notlimited to what is illustrated in FIG. 29. However, it is consideredthat a transmittance spectrum close to this is shown in the color filter3 that is currently used in general. Note that 1 on the vertical axisindicating transmittance does not mean that light is transmitted 100%but indicates, for example, the maximum transmittance in the colorfilter 3.

The cover glass 4 covers and protects the sensor main body 2 and thecolor filter 3.

Here, DBPF 5 is an optical filter formed on the cover glass 4. DBPF 5 isan optical filter that has a transmission characteristic in thevisible-light band, a blocking characteristic in a first wavelength bandadjacent to a long-wavelength side of the visible-light band, and atransmission characteristic in a second wavelength band that is a partin the first wavelength band. Note that an arrangement position of DBPF5 is not limited to the cover glass 4 and arrangement in a differentplace in the imaging sensor 1 may be performed. Also, an arrangementposition of DBPF 5 is not limited to the imaging sensor 1 andarrangement in an optical system that includes a lens 11 and that formsan image on the imaging sensor 1 may be performed.

As indicated by DBPF (solid line) in the graph in FIG. 29, DBPF 5 hashigh transmittance in two bands that are a visible-light band indicatedby DBPF (VR) and an infrared band (second wavelength band) that is in aslightly apart position on a long-wavelength side of the visible-lightband and that is indicated by DBPF (IR). Also, DBPF (VR) as a band withhigh transmittance in the visible-light band is, for example, awavelength band of around 370 nm to 700 nm. Also, DBPF (IR) as a secondwavelength band with high transmittance on an infrared side is, forexample, a band of around 830 nm to 970 nm.

In the present embodiment, a relationship between a transmittancespectrum of each region in the color filter 3 and a transmittancespectrum of DBPF 5 is prescribed as follows.

That is, DBPF (IR) that is the second wavelength band to transmitinfrared light in the transmittance spectrum of DBPF 5 is included in awavelength band A which is illustrated in FIG. 29 and in whichtransmittance of all of the R region, G region, and B region issubstantially the maximum and the transmittance of the regions becomessubstantially the same and in a wavelength band B to transmit light atsubstantially the maximum transmittance in the IR region.

Here, the wavelength band A in which transmittance of the R, G, and Bregions becomes substantially the same is a part where a difference intransmittance of the regions is 10% or smaller in the transmittance.

Note that on a short-wavelength side of the wavelength band A(wavelength band C), the transmittance of the G and B regions becomeslow while the transmittance of the R region becomes substantially themaximum. In DBPF 5, this part with a difference in the transmittance ofthe R, G, and B regions corresponds to apart, in which light is almostblocked and transmittance becomes the minimum in DBPF 5, between DBPF(VR) that is a part with high transmittance in the visible-light bandand DBPF (IR) that is a part with high transmittance in the secondwavelength band arrow.

In such a manner, originally, an output level supposed to be in amountain-like shape in which an output level of a center part in pixelsin one column becomes high. However, as described above, since an outputlevel of IR which level does not exceed a dynamic range and is in amountain-like shape is subtracted from an output level in which anoriginal output level exceeds a dynamic range and is clipped and whichbecomes flat at a pixel saturation level, a part in which an outputlevel is originally the highest is in a state of being recessed incontrast.

In a case where the pixel saturation level is exceeded, an output levelof an image signal of a pixel in each color reaches a peak. At the time,IR is still lower than the pixel saturation level and an output level ofIR becomes higher as luminance becomes higher even after the othercolors exceed the pixel saturation level. That is, even when luminancebecomes higher, an output level of a signal of each of RGB pixels isclipped and does not become higher but an output signal of IR to besubtracted from these becomes higher. Thus, an output level acquired bysubtraction of an output level of IR from an output level of each of RGBbecomes lower as luminance becomes higher. Accordingly, an output levelbecomes low in contrast in a part where an output level is supposed tobe the highest and luminance of the infrared light band. That is, on theinfrared side, transmission of light in a part where the difference inthe transmittance of the R, G, and B regions becomes large is cut andlight is transmitted in the wavelength band A which is on along-wavelength side thereof and in which the transmittance of theregions becomes substantially the maximum and the transmittance becomessubstantially the same.

From the above, in the present embodiment, since a region to transmitlight is not only in the visible-light band but also in the secondwavelength band on the infrared light side in DBPF 5 used instead of aninfrared light cut filter, there is an influence of light passingthrough the second wavelength band in color photographing with visiblelight. However, as described above, the second wavelength band does nottransmit light in a part where transmittance of the R, G, and B regionsis different and only transmits light in a wavelength band in whichtransmittance of the regions becomes substantially the maximum andsubstantially the same.

Also, in a second wavelength low region of DBPF 5, light in a part wheretransmittance of the IR region becomes substantially the maximum istransmitted. Thus, in a case where it is assumed that R, G, B, and IRregions are respectively provided in extremely close four pixels towhich substantially the same light is emitted, in the second wavelengthband, light passes through the R region, the G region, the B region, andthe IR region in substantially a similar manner. As light on an infraredside, light of substantially the same quantity reaches photodiodes ofthe pixels of the imaging sensor main body 2 in the regions includingIR. That is, quantity of light, which passes through the secondwavelength band on the infrared side, in light transmitted through eachof the R, G, and B filters becomes similar to quantity of light passingthrough the IR region. In a case where assumption is made in theabove-described manner, basically, a difference between an output signalof a pixel assumed in the above-described manner which signal is fromthe sensor main body 2 that receives light transmitted through each ofthe R, G, and B filters and an output signal of a pixel assumed in theabove-described manner which signal is from the sensor main body 2 thatreceives light passing through the IR filter becomes an output signal ina visible light part of each of R, G, and B from which partinfrared-side light passing through each of the R, G, and B regions iscut.

Actually, as illustrated in each pattern of the color filter 3 (3 a, 3b, and 3 c), any one of R, G, B, and IR regions is arranged in eachpixel of the sensor main body 2. Since it is likely that quantity oflight in each color which light is emitted to each pixel varies, forexample, it is possible to calculate luminance of each color in eachpixel by using a known interpolation method and to set a differencebetween this interpolated luminance of R, G, and B in each pixel andluminance of IR, which luminance is also interpolated, as luminance ofeach of R, G, and B in each pixel. Note that an image processing methodof removing an infrared light component from luminance of each of thecolors of R, G, and B is not limited to this. Any kind of method can beused as long as the method can finally cut an influence of light, whichpasses through the second wavelength band, from luminance of each of R,G, and B. In any method, since DBPF 5 cuts a part where transmittance ofthe R, G, and B regions is different for more than 10% on an infraredside, that is, a part where transmittance is different for more than apredetermined rate, processing of removing an influence of infraredlight becomes easy in each pixel.

Also, this imaging sensor 1 is used as an imaging sensor in an imagingdevice that can perform both of color photographing and infrared lightphotographing. Generally, it is considered that normal photographing isperformed in the color photographing and the infrared photographing isperformed at night by utilization of illumination of infrared light,which cannot be recognized by a human, without utilization ofillumination of visible light. For example, it is considered that nightphotographing with infrared light by utilization of infrared-lightillumination is performed in various monitoring cameras and the like innight photographing at a place where night illumination is not necessaryor a place that is not preferably illuminated at night. Also,utilization in daytime photographing and night photographing forobservation of a wild animal, or the like is also possible.

In a case where infrared light photographing is used as nightphotographing, infrared-light illumination is necessary since quantityof infrared light is in short at night similarly to visible light. Atransmittance spectrum of DBPF 5 illustrated in FIG. 29 is determined inconsideration of a transmittance spectrum of each of the R, G, B, and IRregions and an emission spectrum of light for infrared-lightillumination such as infrared-light LED for illumination.

In such an imaging sensor, the second wavelength band that transmitslight on an infrared side of DBPF 5 is included, on an infrared side ofeach of the R, G, B, and IR regions, in the wavelength band A in whichtransmittance of each region becomes substantially the maximum andtransmittance of the regions becomes substantially the same and thewavelength band B in which transmittance of the IR region becomessubstantially the maximum. In other words, on a long-wavelength side ofa visible-light band, transmittance of only an R fill part among R, G,and B filters becomes substantially the maximum and transmittance of theG and B regions are not substantially the maximum. Thus, light in a partwhere the transmittance of the R, G, and B regions is not substantiallythe same and is different is cut by DBPF 5.

That is, in each of the R, G, B, and IR regions, light in the secondwavelength band is transmitted on the infrared side. Thus, transmittanceon the infrared side in the regions is substantially the same. When thesame quantity of light to be in the second wavelength band is emitted,transmission quantity in the R, G, B, and IR regions becomes the same.Accordingly, as described above, it is possible to correct a color basedon an output signal from a pixel corresponding to each of the R, G, andB regions and to easily acquire an image in which an influence due toinfrared light passing through a second wavelength band of a color incolor photographing is controlled.

Also, since the second wavelength band is made to correspond to a peakof the emission spectrum of the infrared-light illumination whichspectrum is included in the above-described wavelength band A andwavelength band B, light of the infrared-light illumination can beefficiently used. Also, it is possible to narrow down a width of thesecond wavelength band and to reduce an influence of infrared lightpassing through the second wavelength band in color photographing.

That is, with utilization of DBPF 5, it becomes possible to performaccurate correction by subtracting a value of an IR signal from a valueof each of RGB signals of the imaging sensor 1. Here, an imagingprocessing method in the imaging device will be described before adetail description of the signal processing unit 12.

For example, as expressed in the following, a received light componentof a pixel of each color in the imaging sensor 1 is in a state in whichan IR component is added to a component of each color.R pixel R+IRG pixel G+IRB pixel B+IRIR pixel IR

Thus, as expressed in the following, IR correction of removing the IRcomponent from the received light component of each of the RGB pixelsexcluding the IR pixel is performed.R signal(R pixel output)−(IR pixel output)=(R+IR)−IR=RG pixel(R pixel output)−(IR pixel output)=(G+IR)−IR=GB pixel(R pixel output)−(IR pixel output)=(B+IR)−IR=B

Accordingly, it is possible to remove an IR component, which istransmitted through DBPF 5 and transmitted through the color filter,from the regions of the colors other than IR in the color filter.

However, in the R pixel, the G pixel, and the B pixel, sensitivity withrespect to each light source is different and there is a dynamic rangein each pixel of the imaging sensor 1. Thus, electric charge thatexceeds the dynamic range cannot be read and an output from the imagingsensor is clipped and reaches a peak. That is, when input light exceedsthe dynamic range, an output signal is clipped and cut.

As a result, as described in the following, there is an error in acorrected R signal, G signal, and B signal, and a problem such as anunnatural luminance level (decrease in luminance of highlight part),coloring of highlight, or the like is generated.

FIG. 30 to FIG. 32 are graphs for describing a problem in a case wherean IR component is subtracted from a component of each color in a statein which a dynamic range is exceeded. FIG. 30 is a graph illustrating acase of R, FIG. 31 is a graph illustrating a case of G, and FIG. 32 is agraph illustrating a case of B. In each of the graphs illustrated inFIG. 30 to FIG. 32, a vertical axis indicates an output level of asignal from a pixel of each color in the imaging sensor 1. A horizontalaxis indicates passage of time in an output level of one pixel in theimaging sensor 1 or indicates a position on a column of a pixel (such asposition of each pixel on Y axis). Here, for example, the horizontalaxis is a position of a pixel on a Y axis. Thus, a change in an outputlevel of a signal in accordance with a position on the Y axis in eachpixel of each color is illustrated in each graph. In FIG. 33, an outputlevel of an IR signal with respect to each of the colors illustrated inFIG. 30 to FIG. 32 is illustrated and an output level at a position ofeach pixel on the Y axis is indicated similarly to the case of each ofthe RGB pixels in each of the above-described graphs.

A graph on an upper side in each of FIG. 30 to FIG. 32 indicates anoutput level of a signal in a state in which a dynamic range is exceeddue to a difference in a position on the Y axis. That is, an outputlevel of a signal of each pixel becomes lower after a rise along with achange of a position in a right direction on the Y axis. RGB excludingIR is in a state of exceeding the dynamic range and being clipped. Notethat in each pixel of the imaging sensor 1, electric charge can be readonly up to a pixel saturation level. Electric charge of equal to orhigher than the pixel saturation level cannot be read and an outputlevel becomes a state of being clipped. Also, since an output level ofan IR signal is included in an output level of a signal of each of RGBpixels, these respectively become R+IR, G+IR, and B+IR. The outputlevels of RGB are higher than the output level of the IR signal forsingle output levels of RGB which levels do not include IR. Thus, adynamic range is not exceeded in an IR pixel and a state of exceeding adynamic range is likely to be generated in RGB.

In each of FIG. 30 to FIG. 32, a graph that exceeds a pixel saturationlevel and that is indicated by a dotted line indicates an output levelin a case where clipping is not performed. A graph on a lower side ineach of FIG. 30 to FIG. 32 illustrates a case where an output level ofIR is subtracted from an output level on an upper side which levelincludes IR. Here, in an upward convex part, a case where an outputlevel of an IR signal is subtracted from an output level of each of RGBsignals of when not being clipped which level is indicated by the dottedline described above is illustrated. However, actually, each of RGBsignals of before the subtraction is in a clipped state. Thus, a casewhere an output level of an IR, which output level becomes amountain-like shape due to a difference in a position, is subtractedfrom here becomes a state indicated by a graph on the lower side in astate of being recessed downward from the upward-convex graph on thelower side as indicated by an becomes low in a highlight part. Also, apart where an output level of each of RGB exceeds the pixel saturationlevel is supposed to be white. However, there is a difference in anoutput level in a case where each of RGB becomes a downward convex,whereby the highlight part does not become white and is in a state ofbeing colored.

Thus, when an output level of an IR signal is subtracted from an outputlevel of a signal of each of RGB pixels, in a case where the outputlevel of the signal of each of the RGB pixels is clipped at the pixelsaturation level due to the dynamic range being exceeded, subtraction isperformed after the output level of IR to be subtracted therefrom islowered. That is, as illustrated in FIG. 33 and FIG. 34, in a value ofthe output level of the IR signal which level is subtracted from each ofthe output levels of RGB, a limit value is set with respect to theoutput level of the IR signal, which level is subtracted from the outputlevel of the signal of each of the RGB pixels, according to each of RGBsignals (component) in such a manner that the output level is clippedeven in a case of not exceeding the dynamic range and being lower thanthe pixel saturation level. An output level that becomes equal to orhigher than the limit level is brought into a state of being clipped atthe set limit value (being limited to reach peak).

In a case where an IR signal that is clipped at a limit value set foreach of RGB in such a manner is subtracted from each of the RGB signals,since the IR signal to be subtracted therefrom is clipped at the limitvalue unlike the case illustrated in each of FIG. 30 to FIG. 32, anoutput level of each of the RGB signals does not become low whenluminance becomes high, and is clipped at a signal saturation level ofafter the subtraction as illustrated in FIG. 35(a), FIG. 35(b), and FIG.35(c). This signal saturation level is a value acquired by subtractionof a limit value of the IR signal, which value corresponds to each ofRGB, from the pixel saturation level of each of the RGB signals. Thus, aphenomenon such as a decrease in luminance in the highlight part is notgenerated. Note that a signal saturation level of each of the RGBsignals which level is generated after the subtraction of the IR signalbecomes a level lower than the pixel saturation level since the limitvalue of the IR signal is subtracted from the pixel saturation level ina case where each of the RGB signals exceeds the dynamic range. Also, asdescribed later, the signal saturation level of each of the RGB signalsafter the subtraction of the IR signal varies depending on a color.

Note that a position where an output level of the IR signal is clipped(limit value of IR signal) illustrated in FIG. 34 varies depending oneach of the RGB signals, on which subtraction is performed, and asituation.

That is, with respect to a limit value (clip level) of the output levelof the IR signal of when correction of each of the RGB signals isperformed, an appropriate clip level varies according to each of thecolors of RGB and depending on spectral sensitivity of the imagingsensor 1, a color temperature of a light source, or a kind of a lightsource. Although it is difficult to uniformly calculate this level bycalculation, it is possible to determine the level by measuring anoutput level of each of RGB signals and an output level of an IR signalthat are output from a sensor with respect to each light source (colortemperature). Note that spectral sensitivity of the imaging sensor 1 isdetermined depending on the imaging sensor 1. Also, a color temperatureis determined to some extent depending on a kind of a light source.Thus, it is necessary to calculate the above-described limit value in avalue of an IR signal, which limit value corresponds to each of RGBsignals and at which limit value an output level of IR is clipped, onthe basis of a color temperature.

In a camera, a level of each of the RGB signals varies according to achange in a color temperature of a light source. For example, an Rsignal is increased and a B signal is decreased with respect to a lightsource with a low color temperature.

Also, a B signal is increased and an R signal is decreased with respectto a light source with a high color temperature. As a result, an imagebecomes reddish in a case where a color temperature is low and an imagebecomes bluish in the case where a color temperature is high. Thus,color reproducibility varies depending on a change in a colortemperature of a light source. In order to stabilize this colorreproducibility, white-balance processing (WB) to make levels of the RGBsignals constant is performed.

The white-balance processing is performed by measurement of a colortemperature of a light source from a color signal and by adjustment of again of each of color signals of RGB. Today, a system of performingwhite balance detection from an image signal and performing controlaccording to a result of the detection is generally used. For example, afeedback control loop is configured by an RGB gain adjustment circuit(included in control circuit 21 in FIG. 38) and a white-balancedetection circuit 26 (illustrated in FIG. 38). In the white-balancedetection circuit 26, integration of an R−Y signal, a B−Y signal, or R,G, and B signals is performed. In a case of performing integration ofthe R−Y signal and the B−Y signal, control is performed in such a mannerthat an integration value thereof becomes zero. Also, in a case ofperforming integration of the R, G, and B signals, a gain of each of theRGB signals is control led in such a manner that integration valuesbecomes identical to each other.

Here, it is possible to calculate a ratio of the R signal, G signal, andB signal from the gain of each of the RGB signals and information ofdetermining a color temperature of a light source is acquired. With thisinformation, a limit value (clip level) of an IR signal with respect toeach of the RGB signals in IR correction is determined.

Note that as component systems of a video signal, there are an RGBsystem, in which RGB corresponding to three primary colors arecomponents, and a color difference system using a luminance-colordifference in which RGB are converted into a luminance signal and acolor difference signal. As the color difference system, a system usingY, Cb, and Cr, a system using Y, Pb, and Pr, or the like is known. Y isluminance, Cb and Pb are acquired by multiplication of (B (blue)−Y(luminance)) by a coefficient, and Cr and Pr are acquired bymultiplication of (R (red)−Y (luminance)) by a coefficient, Cb and Pbbeing B−Y signals and Cr and Pr being R−Y signals.

For example, Y, Cb, and Cr are expressed by the following expressionswith respect to R, G, and B.Y=0.299*R+0.587*G+0.114*BCb=0.564*(B−Y)=−0.169*R−0.331*G+0.500*BCr=0.713*(R−Y)=0.500*R−0.419*G−0.081*B

A color difference system is to reconstruct RGB signals into a componentindicating luminance (luminance) and a component indicating a differencebetween two color signals and luminance signals (color difference).Since a human eye hardly realizes deterioration in resolution of acolor, for example, when an information amount of a color difference isreduced to ½ at the time of transmission, a processing amount becomes ⅔of that of RGB.

In the following, a method of setting this clip level (limit value) willbe described with reference to an example.

First, processing of calculating a gain of each of RGB signals in thewhite-balance processing will be described.

In the white balance detection, a region considered to be white is setand the set white region is detected, the white balance detection beingperformed in this region. That is, in a case where each of a B−Y signaland an R−Y signal that are color difference signals in each pixel in theregion considered to be white is integrated, a gain of each of the B−Ysignal and the R−Y signal is adjusted in such a manner that anintegration value thereof becomes zero. Also, a gain of each of the R,G, and B signals is adjusted in such a manner that integration values ofthe RGB signals become identical to each other.

An example of a region considered to be white in the white balancedetection is illustrated in each of FIG. 36(a) and FIG. 36 (b). Asillustrated in FIG. 36 (a), on a B−Y and R−Y plane, a white detectionarea is set in a vicinity of a movement locus due to a color temperatureof a point to be white. Also, as illustrated in FIG. 36(b), a detectionrange of white with respect to a luminance signal level (such as equalto or higher than 70% and lower than 105% of white level) is set. In thecolor difference system, when a color difference signal is in adetection area of white and a luminance signal is in a detection rangeof white, the pixel is in a white region. Each of R, G, and B signals oreach of a B−Y signal and an R−Y signal in the color difference system ofa pixel in this region is integrated.

Alternatively, a gain of each the R, G, and B signals is adjusted insuch a manner that integration values of R, G, and B are identical toeach other.

For example, each of adjusted gains of R, G, and B of when a gain ofeach of the R, G, and B signals is adjusted in such a manner thatintegration values of R, G, and B become identical to each other becomesinformation indicating color temperature information. In thewhite-balance processing, it is possible to acquire each of the RGBsignals of after the white-balance processing by respectivelymultiplying corresponding RGB signals by the gains of RGB which gainsare adjusted in this manner.

That is, the RGB signals of after the white balancing are(R signal of after WB)=(gain of R signal)×(R signal of before WB)(G signal of after WB)=(gain of G signal)×(G signal of before WB)(B signal of after WB)=(gain of B signal)×(B signal of before WB).

Next, a method of calculating a limit value as a correction value of avalue of an IR signal subtracted from each of signals of R (R+IR), G(G+IR), and B (B+IR) by using each of the gains of RGB which gains areadjusted by the white-balance processing as described above will bedescribed. Note that a limit value of an IR signal subtracted from eachof the signals of R (R+IR), G (G+IR), and B (B+IR) is a value thatbecomes an upper limit in a case where each of the color signals of R(R+IR), G (G+IR), and B (B+IR) is in a state of exceeding a pixelsaturation level and is clipped at the pixel saturation level and in acase where a value of the IR signal is subtracted from each of the colorsignals.

A limit value (clip level) of the IR signal which value corresponds toeach of the RGB signals is expressed by the following expressions when asaturation level of a pixel is Lsat, a ratio of an R signal to an IRsignal of when white is imaged at a certain color temperature is Kr, anda limit value (clip level) of the IR signal in the R signal is Lclip−R.Lsat=Lclip−R+Kr*Lclip−RLclip−R=Lsat/(1+Kr)

Similarly, when a ratio of a G signal to an IR signal is Kg, a ratio ofa B signal to an IR signal is Kb, a limit value (clip level) of the IRsignal in the G signal is Lclip−G, and a limit value (clip level) of theIR signal in the B signal is Lclip−B,Lclip−G=Lsat/(1+Kg)Lclip−B=Lsat/(1+Kb).

Here, as described above, when a ratio of the G signal to the R signalis Kg/r and a ratio of the B signal to the R signal is Kb/r in imagingof white,(R signal of after WB)=(gain of R signal)×(R signal of before WB)(G signal of after WB)=(gain of G signal)×(G signal of before WB)(B signal of after WB)=(gain of B signal)×(B signal of before WB).

Accordingly,(R signal of before WB)=(R signal of after WB)/(gain of R signal)(G signal of before WB)=(G signal of after WB)/(gain of G signal)(B signal of before WB)=(B signal of after WB)/(gain of B signal).

Also, an R signal, a G signal, and a B signal of after WB are identicalwith respect to white. Thus,Kg/r=(gain of R signal)/(gain of G signal)Kb/r=(gain of R signal)/(gain of B signal).Also,Kg=Kr×Kg/rKb=Kr×Kb/r.Thus,Lclip−G=Lsat/(1+Kr×Kg/r)Lclip−B=Lsat/(1+Kr×Kb/r).

Accordingly, it is possible to calculate a red clip level Lclip−R, agreen clip level Lclip−G, and a blue clip level Lclip−B of an IR signal.

Also, Kb/r can be used as a parameter indicating a color temperature. Ateach color temperature, when Kr with respect to Kb/r is measured and ispreviously recorded in a memory or the like, it is possible to determinea clip level of an IR signal for correction of each color according toKb/r and Kr with respect to this Kb/r on the basis of a gain acquired bythe white balance detection and the above-described expressions.

Also, in a case where IR correction (calculation of clip level of IRsignal) is performed in the above manner, signal saturation levels inhighlights of the RGB signals (level of signal in subtraction ofcorrected IR signal (limit value: clip level) from RGB signal thatreaches pixel saturation level and is clipped) are not always identicalto each other. This becomes a cause of a colored highlight anddeterioration in image quality. Thus, as illustrated in FIG. 37(a), FIG.37(b), and FIG. 37(c), correction is performed after the white balancingin such a manner that highlight parts of RGB are clipped at the samelevel (RGB clip level). Accordingly, a colored highlight andunnaturalness in gradation of luminance are not generated.

In each of FIG. 37(a) to FIG. 37(c), a vertical axis is an output levelof when an IR signal is subtracted from each of signals of RGB (R+IR,G+IR, and B+IR) and a horizontal axis indicates, for example, a positionof a pixel in a Y axis direction on the imaging sensor 1 or passage oftime of one pixel. In each graph in FIG. 37(a) to FIG. 37(c), in a casewhere a color temperature of a light source does not vary and a cliplevel (limit value) of each of RGB which level is calculated in theabove-described manner is constant in a state in which each of the RGBsignals reaches a pixel saturation level and in a case where an IRsignal that is limited (clipped) at a limit value (clip level) issubtracted from each of the RGB signals at the pixel saturation level,since the pixel saturation level of each of the RGB signals is constantand limit values of the IR signal which values respectively correspondto RGB are constant, the RGB signals of after the subtraction are in aconstant state, that is, in a constant state at the signal saturationlevels. However, as illustrated in FIG. 37(a) to FIG. 37(c), the signalsaturation levels of RGB are different from each other. In this case,highlight parts do not become white since output levels of the RGBsignals are different. Thus, the signal saturation levels of the RGBsignals are adjusted to a common RGB clip level with a signal saturationlevel of the R signal, which signal saturation level is the lowest, asthe common RGB clip level (R signal saturation level) of the RGBsignals.

That is, as illustrated in FIG. 37(a), by subtraction of an IR signal,in which a clip level (limit value) corresponding to an R signal is set,from the R signal, a signal saturation level (clip level) generated withrespect to the R signal of after the subtraction is set as a reference.Among the signal saturation levels of the RGB signals of after thesubtraction of the IR signal, that of the R signal is the lowest. Thus,as illustrated in FIG. 37 (b) and FIG. 37 (c), the signal saturationlevel of the G signal and the signal saturation level of the B signal ofafter the subtraction of the IR signal are lowered to be adjusted to theRGB clip level that is identical to the signal saturation level of the Rsignal with the R signal as a reference as described above, whereby thesignal saturation levels of the RGB signals of after the IR signalsubtraction are adjusted to each other. Accordingly, high levels of thesignals in highlight parts become the same and it is possible to preventthe highlights from being colored.

Note that in the above-described configuration, an IR signal issubtracted from each of an R signal, a G signal, and a B signal afterbeing clipped. However, when the IR signal becomes equal to or higherthan a level at which the above R signal, G signal, and B signal aresaturated (above-described clip level of IR signal), the IR signal maybe subtracted from the R signal, the G signal, and the B signal after again is lowered by a multiplier. In such a manner, unnaturalness inluminance gradation of the R signal, the G signal, and the B signal inthe highlight parts may be prevented by a configuration with which asubtraction amount is controlled. Also, coloring may be prevented byprocessing of generating luminance signals from the RGB signals,decreasing gains of an R−Y signal and a B−Y signal at a level equal toor higher than that of a part where coloring is caused due to adifference in the signal saturation levels of the RGB signals, andremoving the colors.

FIG. 38 is a block diagram illustrating signal processing in the signalprocessing unit 12 (illustrated in FIG. 26) of the imaging device 10(illustrated in FIG. 26). Usually, in an RAW output, with respect to anoutput signal of each of pixels of R, G, B, and IR from the imagingsensor 1 (input signal from imaging sensor 1 in this signal processing),R, G, B, and IR are output line-sequentially or point-sequentially.Thus, for example, a synchronization circuit (not illustrated) of eachcolor signal is provided in an input unit of an RAW signal from theimaging sensor 1.

In this case, in the synchronization circuit, R, G, B, and IR signalsare respectively converted into image data in which all pixels areexpressed in red R, image data in which all pixels are expressed ingreen G, image data in which all pixels are expressed in blue B, andimage data in which all pixels are expressed in infrared IR byinterpolation processing in image data for each frame. In other words,all of the pixels are brought into a state in which the R, G, B, and IRsignals are output. Note that as an interpolation processing method, aknown method can be used.

That is, the signal processing unit 12 includes a synchronizationcircuit (not illustrated) for each of sensor outputs of R+IR, G+IR,B+IR, and IR, and the sensor outputs of R+IR, G+IR, B+IR, and IR in FIG.38 are signals of after passing through the synchronization circuits.

In the signal processing unit 12, limiters 20 r, 20 g, and 20 b to clipan IR signal, which is subtracted from each of RGB signals, at a cliplevel (limit value) determined for each of RGB, multipliers 22 r, 22 g,and 22 b to multiply the IR signal output from each of the limiters 20r, 20 g, and 20 b by a correction value and to perform correction,subtractors 23 r, 23 g and 23 b to respectively subtract the clipped IRsignals output from the multipliers 22 r, 22 g, and 22 b from thesignals of R+IR, G+IR, and B+IR, multipliers 24 r, 24 g, and 24 b torespectively multiply the RGB signals, from which IR signals aresubtracted, by gains of RGB for white balancing, and limiters 25 r, 25g, and 25 b to adjust signal saturation levels of the RGB signals fromwhich the IR signals are subtracted and on which the white balancing isperformed are provided.

Also, the signal processing unit 12 includes a control circuit 21 thatrespectively calculates clip levels (limit value) of IR signals withrespect to the RGB signals and outputs these to the limiters 20 r, 20 g,and 20 b. Also, the control circuit 21 outputs a correction value toeach of the multiplier 22 r, 22 g, and 22 b, outputs gains of RGBcalculated for the white balancing to the multipliers 24 r, 24 g and 24b, and outputs an RGB clip level (R signal saturation level) to adjustthe signal saturation levels of the RGB signals to the limiters 25 r, 25g and 25 b. Also, the signal processing unit includes a white-balancedetection circuit 26 to calculate gains for the white balancing from RGBoutput signals from the signal processing unit 12.

An IR signal as a sensor output of IR is transmitted to each of thelimiter for R 20 r, the limiter for G 20 g, and the limiter for B 20 bprovided in the signal processing unit 12 and is clipped at a limitvalue (clip level) for the IR signal which value is set for each coloras described above.

In this case, the above-described red clip level Lclip−R, green cliplevel Lclip−G, and blue clip level Lclip−B of the IR signal, whichlevels are calculated in the control circuit 21, are respectively outputto the corresponding limiter for R 20 r, the limiter for G 20 g, and thelimiter for B 20 b and respectively become limit values of the limiterfor R 20 r, the limiter for G 20 g, and the limiter for B 20 b.Accordingly, an output level of the IR signal, which level exceeds theclip level (limit value), is clipped at the clip level by each of thelimiters 20 r, 20 g, and 20 b.

Also, values output from the limiter for R 20 r, the limiter for G 20 g,and the limiter for B 20 b are IR signals to be subtracted from the RGBsignals. These IR signals are multiplied by the correction values, whichare output from the control circuit 21, by the multipliers 22 r, 22 g,and 22 b. In the present configuration, IR components included in an Rpixel, a G pixel, and a B pixel are substantially at the same level withthat in an IR pixel. However, there is a possibility that some errorsare generated in a signal level due to a difference in openings of the Rpixel, the G pixel, and the B pixel, a variation in an amplifier gaininside a sensor, or the like. That is, for example, values output fromthe limiters 20 r, 20 g, and 20 b have a tendency of becoming slightlylarger than necessary values and are corrected by the multipliers 22 r,22 g, and 22 b by multiplication by correction values according to theRGB signals.

The subtractors 23 r, 23 g, and 23 b subtract IR signals, which areclipped and corrected in such a manner and which respectively correspondto RGB, from the signals of R+IR, G+IR, and B+IR. The multipliers 24 r,24 g, and 24 b perform white balancing by multiplying the signals, whichare output from the subtractors 23 r, 23 g, and 23 b, by gains of RGBwhich gains are calculated in white-balance processing. Into themultipliers 24 r, 24 g, and 24 b, gains of RGB which gains arecalculated by the control circuit 21 on the basis of RGB signalsdetected in the white-balance detection circuit 26 are respectivelyinput. Also, RGB signals output from the multipliers 24 r, 24 g, and 24b are clipped when clipped IR signals are subtracted therefrom.

That is, when an IR signal limited at a limit value in theabove-described manner is subtracted in a state in which each of RGBsignals is clipped at a pixel saturation level, clipping at a signalsaturation level that is a value acquired by subtraction of the limitvalue from the pixel saturation level is performed. When signalsaturation levels of the RGB signals are different from each other, ahighlight part is colored. Thus, in order to adjust signal saturationlevels of RGB, limit values are set in the limiters 25 r, 25 g, and 25 bin such a manner that RGB clip levels of the RGB signals become thesame. Here, clip levels of the G signal and the B signal are adjustedwith a signal saturation level of the R signal as a limit value of anRGB level.

The signal processing in the above-described manner will be describedwith an R (R+IR) signal as an example. An R signal output from theimaging sensor 1 is interpolated by a synchronization circuit and is setas an R signal of each pixel used as an image in the imaging sensor 1.Similarly, a G signal, a B signal, and an IR signal are processed in thesynchronization circuit and are set as a G signal, a B signal, and an IRsignal of each pixel used as an image.

The IR signal is transmitted to the limiter 20 r. To the limiter 20 r,Lclip−R that is a limit value of an IR signal for an R signal and iscalculated in the above-described manner in the control circuit 21 isoutput. This becomes a limit value of the limiter 20 r. Thus, the IRsignal passing through the limiter 20 r is clipped in such a manner asto reach a peak at the limit value. That is, an output level of the IRsignal exceeding the limit value becomes the limit value.

The IR signal clipped at the limit value in such a manner is transmittedto the multiplier 22 r, corrected by being multiplied by a correctionvalue calculated (or stored) in the control circuit 21, and transmittedto the subtractor 23 r. Into the subtractor 23 r, the above-describedsynchronized R signal is input and the IR signal that is limited andcorrected in the above-described manner is input. A value of the IRsignal is subtracted from that of the R signal. Then, the R signal fromwhich the IR signal is subtracted is transmitted to the multiplier 24 r.Into the multiplier 24 r, the R signal and a gain of the R signal, whichgain is calculated by the above-described white-balance detectioncircuit 26 and control circuit 21, are input. The R signal is multipliedby the gain.

An R signal on which the white-balance processing is performed is outputfrom the multiplier 24 r. This R signal is input into the limiter 25 r.To the limiter 25 r, the above-described RGB clip level is transmittedfrom the control circuit 21. With this as a limit value, an R signalexceeding the RGB clip level is clipped. However, in this signalprocessing unit 12, the RGB clip level is identical to an R signalsaturation level. The R signal from which the IR signal is subtracted isalready in a state of being clipped at the R signal saturation level.Thus, the processing in the limiter 25 r is not necessarily requiredwith respect to the R signal. Note that the G signal and the B signalare clipped at the RGB clip level by the limiter 25 g and 25 b with theRGB clip level, which is identical to the R signal saturation level, asa limit value.

The G signal and the B signal are processed in a similar manner and eachof the processed RGB signals is output as an output signal 14 of a colorimage of visible light as illustrated in FIG. 1. Also, the IR signal isoutput as an output signal 15 of an infrared-light image after passingthrough the synchronization circuit. Note that the IR signal output asan infrared image signal is not limited by the above-described limitvalue.

A block diagram illustrated in FIG. 39 is a modification example of theblock diagram illustrated in FIG. 38. A different between the blockdiagram illustrated in FIG. 39 and the block diagram illustrated in FIG.38 is that multipliers 24 r, 24 g and 24 b for white balancing arearranged after limiters 25 r, 25 g and 25 b to adjust clip levels of RGBsignals, from which IR signals are subtracted, in FIG. 39 while thelimiters 25 r, 25 g, and 25 b to adjust signal saturation levels of RGBsignals, from which IR signals are subtracted, are arranged after themultipliers 24 r, 24 g, and 24 b for white balancing in the blockdiagram in FIG. 38.

Accordingly, while signal saturation levels of the RGB signals on whichwhite balancing is performed are adjusted in FIG. 38, white-balanceprocessing is performed after adjustment of signal saturation levels ofRGB signals in FIG. 39.

According to such an image processing system and an imaging processingmethod, it is possible to prevent a decrease in luminance of a highlightpart and to prevent the highlight part from being colored instead ofbecoming white as described above in a camera (imaging device) thatmakes it possible to perform both of color photographing with visiblelight and photographing with infrared light by using DBPF 5 withoutswitching whether to use an infrared cut filter.

Next, a seventh embodiment will be described. In the above-describedsixth embodiment, a clip level (control value) in a case where an RGB-IRcolor filter is used has been described. In the present embodiment, aclip level of RGB-C will be described.

In an imaging sensor 1 of RFB-IR, an IR signal only includes an IRcomponent while an IR component is included in each of RGB signals.Thus, even when each of the RGB signals is saturated, the IR signal isnot saturated. Here, when an IR component a level of which becomeshigher even after the RGB signals are saturated is removed, a decreaseto a saturation level or lower is caused by removal of the IR componentafter saturation of RGB as described above. Also, basically, a value ofRGB of after the saturation becomes smaller as light quantity of IRbecomes larger.

On the other hand, in an imaging sensor 1 of RGB-C, C=R+G+B in avisible-light band. Thus, in a case where a range is adjusted to RGB, apixel of C reaches a saturation level as illustrated in FIG. 40.

In this case, when the pixel of C reaches the saturation level, C<R+G+Bis acquired in the visible-light band. Accordingly, in a case where IR′is acquired by calculation as described above, IR′=IR+(R+G+B−C)/2 isacquired. Since (R+G+B−C) becomes larger than 0 when C reaches thesaturation level, IR′ acquired by calculation>actual IR, as illustratedin FIG. 41.

In such a manner, when the IR′ component acquired by the calculationfrom R, G, B, and C is removed in a case where C reaches the saturationlevel, a signal level with respect to C reaches a peak at the saturationlevel but an IR component can become higher even after C reaches thesaturation level. Also, as illustrated in FIG. 41, IR′>IR is acquiredafter C is saturated. Thus, when this is removed from saturated C, asignal level becomes lower as a level of actual light quantity becomeshigher as illustrated in FIG. 42. Also, after C is saturated, IR′>IR isalso acquired with respect to signal levels of RGB other than C. Thus,as a level of actual light quantity becomes higher, the signal levelsmay become lower although a degree is not as large as that of C.

Thus, in the seventh embodiment, each signal is clipped in such a manneras not to be equal to or larger than a predetermined value and an IRsignal is separated from each of the RGB signals.

First, a signal level of R/G/B/W/IR is detected.

Then, clip levels in R, G, B, and W are calculated in the followingmanner.(W clip level)=(calculated by saturation level of W pixel (C pixel))(R clip level)=(W clip level)(R level+IR level)/(W level+IR level)(G clip level)=(W clip level)(G level+IR level)/(W level+IR level)(B clip level)=(W clip level)(B level+IR level)/(W level+IR level)

In a case where a signal level of W is detected, the W clip level iscalculated on the basis of a saturation level that does not becomehigher with this signal level as a peak. The clip levels of RGB arecalculated by the above-described expressions. The clip levels arecalculated when the W level reaches the saturation level.

In a case where such clip processing is performed, the above-describedseparation device 51 is in a manner illustrated in a block diagram inFIG. 43. Signal levels of R, G, B, and W of each pixel are separated byan interpolation method or the like in a color separation device 61 andtransmitted to a level detection/clip-level calculation device 63 and aclip processing device 62. In the level detection/clip-level calculationdevice 63, clip levels of R, G, B, and W are calculated on the basis ofthe above-described expressions and detected levels of R, G, B, and W.Each of the clip levels of R, G, B, and W calculated in the clip-levelcalculation device 63 is transmitted to the clip processing device 62.Processing of performing clipping is performed in a case where signallevels of R, G, B, and W input into the clip processing device 62 exceedthe clip levels. The signal levels of R, G, B, and W output from theclip processing device 62 are input into an IR correction/IR generationdevice 64. In the IR correction/IR generation device 64, an IR signal isremoved from each of R, G, B, and W signals and an IR signal isgenerated.

As illustrated in FIG. 44, the clip-level calculation device 63calculates an IR signal level in an IR matrix device 66 on the basis ofsignal levels of R, G, B, and W separated in the color separation device61, inputs the IR signal and the signal levels of R, G, B, and Wincluding IR components into an IR correction device 67, and performs IRcorrection of removing an IR signal level from each of the signal levelsof R, G, B and W.

Into a level detection device 68, the signal level of IR is input fromthe IR matrix device 66 and each of the signal levels of R, G, B, and Wfrom which IR components are removed is input from the IR correctiondevice 67.

Each of the signal levels of R, G, B, W, and IR detected in the leveldetection device 68 is input into a clip-level calculation device 69. Asdescribed above, a clip level of each of the signal levels of R, G, B,and W is calculated in the clip-level calculation device 69.

In the clip processing device 62, on the basis of each of the cliplevels of R, G, B, and W which levels are calculated in such a manner,each of the signal levels reaches a peak at the clip level asillustrated in FIG. 45 in a case where each of the signal levels of R,G, B, and W exceeds the clip level.

Similarly to the above-described third embodiment, on the basis of eachof the R, G, B, and W signals clipped in such a manner, an IR componentis calculated and the IR component is separated from each of the R, G,B, and W signals including the IR components. Here, an IR signalacquired by calculation as illustrated in FIG. 46 is also clipped andreaches a peak at a signal level of IR of when W reaches a saturationlevel. Accordingly, a state of being clipped at a clip level afterremoval of an IR component is kept even after an IR component is removedfrom each of the signal levels of R, G, B, and W as illustrated in FIG.47.

In this case, a luminance signal calculated from each of the signallevels of R, G, B, and W is also in a state of being clipped. Thus, byusing a modification example of a separation device 51 to perform colorseparation, IR separation, and IR correction as illustrated in FIG. 48,it is possible to prevent a luminance level from being clipped and togradate the luminance level even after a pixel of W reaches a saturationlevel.

As illustrated in FIG. 48, in this modification example of theseparation device 51, signal levels of R, G, B, and W of each pixel areseparated by an interpolation method or the like in a color separationdevice 61 and transmitted to a level detection/clip-level calculation/IRsignal generation device 63 and a clip processing device 62. In thelevel detection/clip-level calculation device 63, clip levels of R, G,B, and W are calculated on the basis of the above-described expressionsand detected levels of R, G, B, and W. Each of the clip levels of R, G,B, and W calculated in the clip-level calculation device 63 istransmitted to the clip processing device 62. Processing of performingclipping is performed in a case where signal levels of R, G, B, and Winput into the clip processing device 62 exceed the clip levels. Thesignal levels of R, G, B, and W output from the clip processing device62 are input into a first IR correction/IR generation device 64. In thefirst IR correction/IR generation device 64, an IR signal is removedfrom each of R, G, B, and W signals to be clipped and an IR signal isgenerated. The RGB signals calculated in the first IR correction/IRgeneration device 64 are used as color signals such as color differencesignals.

Also, a second IR correction: an IR generating device 65 of thismodification example is included. Into a second IR correction/IRgeneration device 65, R, G, B, and W signals that are not to be clippedare input from the color separation device 61. Then, IR generation andIR correction are performed and RGB signals from which IR components areremoved are output. Here, the RGB signals are less likely to reachsaturation levels compared to a W signal and are less likely to reachtheir peaks. As illustrated in FIG. 49, by using these RGB signals forcalculation of luminance, it is possible to prevent a luminance levelfrom being clipped and to slightly gradate the luminance even in a statein which the luminance is high.

Next, an eighth embodiment will be described. In the eighth embodiment,a clip level of each of R, G, B, and W signals is determined similarlyto the seventh embodiment. Unlike the seventh embodiment, clip levels ofan IR signal and a W signal are determined without determination of aclip level of each of R, G, and B signals in the eighth embodiment. Asillustrated in FIG. 50, IR correction is performed by an IR signalclipped at a control value by limit processing.

A W clip level is set on the basis of a saturation level of W.(W clip level)=(calculated by saturation level of W pixel (C pixel))

An IR clip level is determined on the basis of the following expression.(IR clip level)=(W clip level)(IR level)/(W level+IR level)

Accordingly, as illustrated in FIG. 50, in a case where a signal levelof IR which level is clipped in the above-described manner is removedfrom a signal level of W, which reaches a saturation level, when a Wsignal is substantially saturated, the signal level of W is clipped at asignal level lower than the saturation level.

When it is assumed that signal levels of RGB do not reach saturationlevels in a range of a condition of use in a case where correction ofremoving a clipped IR signal from each of the signal levels of RGB, thesignal levels of RGB can become higher even after the IR signal reachesa clip level in the removal of the IR signal as illustrated in FIG. 51.Thus, each of the RGB signals is not clipped and can become higher.

In a separation device 51 of the eighth embodiment which device isillustrated in FIG. 52, signal levels of R, G, B, and W of each pixelare separated by an interpolation method or the like in a colorseparation device 61 and separated R, G, B, and W signals are output toan IR signal generation device 71 and an IR correction device 64. In theIR signal generation device 71, a clip level of a generated IR signal iscalculated and an IR signal that is clipped in a case where the cliplevel is exceeded is output to the IR correction device 64. In the IRcorrection device 64, the IR signal is removed from each of the signallevels of R, G, B and W.

As illustrated in FIG. 53, in the IR signal generation device 71, eachof the R, G, B, and W signals is transmitted from the color separationdevice 61 to an IR correction device 72 and an IR matrix device 73. AnIR signal generated in the IR matrix device 73 is output to the IRcorrection device 72 and a limit processing device 74. In the IRcorrection device 72, IR correction of removing the IR signal from eachof the signal levels of R, G, B, and W is performed, the corrected R, G,B, and W signals and the IR signal are transmitted to a level detectiondevice 75, the R, G, B, and W signals and the IR signal are transmittedto a limit level calculation device 76, and a limit level (clip level)of the IR signal is input into the limit processing device 74. Into thelimit processing device 74, an IR signal level is input from the IRmatrix device 73. In a case of exceeding the limit level, the IR signallevel is clipped at the limit level.

Similarly to the seventh embodiment, in such an eighth embodiment, it ispossible to prevent a situation in which signal levels of R, G, B, and Wbecome lower in a condition in which these supposed to become higher andto gradate each of signal levels of RGB even in a state in which asignal level of W is saturated. Accordingly, it is also possible togradate a luminance signal even in a situation in which the signal levelof W is saturated.

Note that an expression for generation of a luminance signal isY=(Kr*R+Kg*G+Kb*B+Kw*W)+Kir*IR.

It is possible to control sensitivity by changing a ratio of IR. Notethat in a case where IR illumination is used at night, there is novisible signal and there are only IR signals in all pixels. In thiscase, it is possible to generate a luminance signal with highsensitivity in high resolution by turning off IR correction andgenerating the luminance signal from signals of all pixels. In a casewhere IR illumination is embedded in a camera, signal processing isswitched along with this.

An expression for generating luminance in IR illumination in this caseisY=Kr*R+Kg*G+Kb*B+Kw*W(IR correction OFF).

Note that in each pixel, quantity of received light from infraredillumination which light passes through a second wavelength band of DBPF5 and each filter becomes larger than quantity of received light in avisible-light band. Thus, basically, each pixel is in a state in which asignal level of infrared light is output.

REFERENCE SIGNS LIST

-   1 imaging sensor-   2 imaging sensor main body-   3 color filter (filter)-   3 a color filter (filter)-   3 b color filter (filter)-   3 c color filter (filter)-   5 DBPF (optical filter: filter)-   A third wavelength band-   B fourth wavelength band-   IR first wavelength band-   DBPF (IR) second wavelength band-   DBPF (VR) visible-light band

The invention claimed is:
 1. An imaging sensor comprising: an imagingsensor main body in which a light-receiving element is arranged in eachpixel; and a filter including: a color filter in which a plurality kindsof filter parts are arranged in a predetermined array in a mannercorresponding to an array of pixels of the imaging sensor body, theplurality of kinds of filter parts having different spectraltransmission characteristics in response to wavelengths in avisible-light band; and an optical filter which has: i) a transmissioncharacteristic in the visible-light band; and ii) a light blockingcharacteristic in a first wavelength band adjacent to a longerwavelength side of the visible-light band, except for a secondwavelength band for light transmission, the second wavelength band beinga predetermined range having a lower limit and an upper limit within thefirst wavelength band; wherein: the color filter has a third wavelengthband, in which transmittances of the plurality of kinds of filter partsare approximate to one another, on a longer wavelength side than thevisible-light band; and the spectral transmission characteristics of theoptical filter and the spectral transmission characteristics of theplurality of kinds of filter parts in the color filter are set in amanner such that the second wavelength band is included in the thirdwavelength band.
 2. The imaging sensor according to claim 1, wherein inthe third wavelength band, a transmittance difference among theplurality of kinds of filter parts is 20% or less.
 3. The imaging sensoraccording to claim 1, wherein the plurality of kinds of filter partsconsist of: i) one kind of the filter parts having a light transmissioncharacteristic in substantially a whole wavelength band of thevisible-light band, or having a blocking characteristic in substantiallya whole wavelength band of the visible-light band; and ii) the otherkinds of the filter parts having transmission characteristics in limitedwavelength bands corresponding to respective colors in the visible-lightband.
 4. The imaging sensor according to claim 3, wherein: the one kindof the filter parts has a blocking characteristic in substantially thewhole visible-light band, and has a transmission characteristic in afourth wavelength band on the longer wavelength side than thevisible-light band; and a transmission characteristic of the opticalfilter and the spectral transmission characteristic of each of theplurality of kinds of filter parts are set in such a manner that thesecond wavelength band of the optical filter is included in the fourthwavelength band.
 5. The imaging sensor according to claim 3, wherein:the one kind of the filter parts has a transmission characteristic insubstantially a whole visible-light band and has a transmissioncharacteristic in a fourth wavelength band on the longer wavelength sidethan the visible-light band; and the transmission characteristic of theoptical filter and the spectral transmission characteristic of each ofthe plurality of kinds of filter parts are set in such a manner that thesecond wavelength band of the optical filter is included in the fourthwavelength band.
 6. The imaging sensor according to claim 3, wherein: inthe color filter, four kinds of the filter parts of red, blue, and greenwith transmission characteristics in the limited wavelength bandscorresponding to the respective colors in the visible-light band and ofa fourth filter part with a blocking or transmission characteristic insubstantially a whole wavelength band in the visible-light band arearranged in a basic array with four rows and four columns; the samekinds of the filter parts are arranged separately so as not to beadjacent to each other in a row direction and a column direction; oneeach of the red, the blue, the green and the fourth filter parts arearranged in each column; and two each of two kinds of the filter partsamong the red, the blue, the green, and the fourth filter parts arearranged in every other row.
 7. The imaging sensor according to claim 3,wherein: the color filter includes eight green filter parts, four redfilter parts, two blue filter parts and two fourth filter parts amongthe four kinds of the filter parts of red, blue, and green withtransmission characteristics in the limited wavelength bandscorresponding to the respective colors in the visible-light band and ofa fourth filter part with a blocking or transmission characteristic insubstantially a whole wavelength band in the visible-light band; theeight green filter parts, the four red filter parts, the two blue filterparts and the two fourth filter parts are arranged in a basic array withfour rows and four columns; and the same kinds of the filter parts arearranged separately so as not to be adjacent to each other in a rowdirection and a column direction.
 8. The imaging sensor according toclaim 3, wherein: the color filter includes eight green filter parts,four fourth filter parts, two red filter parts and two blue filter partsamong the four kinds of the filter parts of red, blue, and green withtransmission characteristics in the limited wavelength bandscorresponding to the respective colors in the visible-light band and ofa fourth filter part with a blocking or transmission characteristic insubstantially a whole wavelength band in the visible-light band; theeight green filter parts, the four fourth filter parts, the two redfilter parts and the two blue filter parts are arranged in a basic arraywith four rows and four columns; and the same kinds of the filter partsare arranged separately so as not to be adjacent to each other in a rowdirection and a column direction.
 9. The imaging sensor according toclaim 1, further comprising: a signal separating/outputting device to:separate a signal, which is sequentially inputted from each pixel of theimaging sensor main body, into: an image signal in the visible-lightband, which is free from a signal in a wavelength band on the longerwavelength side than the visible-light band, and an infrared imagesignal on the longer wavelength side than the visible-light band, andperform an output thereof.
 10. An imaging device comprising: an imagingsensor in which a light-receiving element is arranged in each pixel; acolor filter in which a plurality of kinds of filter parts are arrangedin a predetermined array in a manner corresponding to an array of pixelsof the imaging sensor, the plurality of kinds of filter parts havingdifferent spectral transmission characteristics in response towavelengths in a visible-light band; an optical system including a lensto form an image on the imaging sensor; an optical filter that has: (i)a transmission characteristic in the visible-light band, and (ii) alight blocking characteristic in a first wavelength band adjacent to alonger-wavelength side of the visible-light band, except for a secondwavelength band for light transmission, the second wavelength band beingpredetermined range having a lower limit and an upper limit within thefirst wavelength band; a signal processing device that is capable ofprocessing a signal output from the imaging sensor and of outputting avisible image signal or an infrared image signal, wherein: a spectraltransmission characteristic of the optical filter and the spectraltransmission characteristic of each of the plurality of kinds of filterparts in the color filter are set in such a manner that: the secondwavelength band of the optical filter is included in a third wavelengthband in which transmittances of the plurality of kinds of filter partsare approximate to one another on a longer wavelength side than thevisible-light band.
 11. The imaging device according to claim 10,wherein in the third wavelength band, a transmittance difference amongthe plurality of kinds of filter parts of different colors is 20% orless.
 12. The imaging device according to claim 10, wherein: theplurality of kinds of the filter parts consist of: i) one kind of thefilter parts having a light transmission characteristic in substantiallya whole wavelength band of the visible-light band, or having a blockingcharacteristic in substantially a whole wavelength band of thevisible-light band; and ii) the other kinds of the filter parts havingtransmission characteristics in limited wavelength bands correspondingto respective colors in the visible-light band.
 13. The imaging deviceaccording to claim 12, wherein: the one kind of the filter parts has ablocking characteristic in substantially a whole visible-light band andhas a transmission characteristic in a fourth wavelength band on thelonger wavelength side than the visible-light band; and a spectraltransmission characteristic of the optical filter and the spectraltransmission characteristic of each of the plurality of kinds of filterparts are set in such a manner that the second wavelength band of theoptical filter is included in the fourth wavelength band.
 14. Theimaging device according to claim 12, wherein: the one kind of thefilter parts has a transmission characteristic in substantially a wholevisible-light band and has a transmission characteristic in a fourthwavelength band on the longer wavelength side than the visible-lightband; and the spectral transmission characteristic of the optical filterand the spectral transmission characteristic of each of the plurality ofkinds of filter parts are set in such a manner that the secondwavelength band of the optical filter is included in the fourthwavelength band.
 15. The imaging device according to claim 13, furthercomprising: an infrared illumination unit for producing an infraredillumination during imaging of an infrared image, wherein: setting isperformed in such a manner that: (i) a fifth wavelength band, which is awavelength band of infrared light emitted from the infrared illuminationunit, is included in the third wavelength band and the fourth wavelengthband; and (ii) the second wavelength band of the optical filter issubstantially overlapped with the fifth wavelength band.
 16. The imagingdevice according to claim 14, further comprising: an infraredillumination unit for producing an infrared illumination during imagingof an infrared image, wherein: setting is performed in such a mannerthat: (i) a fifth wavelength band, which is a wavelength band ofinfrared light emitted from the infrared illumination unit, is includedin the third wavelength band and the fourth wavelength band; and (ii)the second wavelength band of the optical filter is substantiallyoverlapped with the fifth wavelength band.
 17. The imaging deviceaccording to claim 10, wherein: the color filter is provided on theimaging sensor; and the optical filter is provided on a surface of thelens constituting the optical system.
 18. The imaging device accordingto claim 10, wherein: the color filter and the optical filter areprovided on the imaging sensor.
 19. An imaging method for use whenoperating an imaging device, the imaging device comprising: an imagingsensor in which a light-receiving element is arranged in each pixel; acolor filter in which a plurality of kinds of filter parts are arrangedin a predetermined array in a manner corresponding to an array of pixelsof the imaging sensor, the plurality of kinds of the filter parts havingdifferent spectral transmission characteristics in response towavelengths in a visible-light band, and each filter part having a thirdwavelength band in which transmittances of each filter part areapproximate to one another on a longer wavelength side than the visiblelight band; and an optical filter that has: (i) a transmissioncharacteristic in the visible-light band; and (ii) a light blockingcharacteristic in a first wavelength band adjacent to a longerwavelength side of the visible-light band, except for a secondwavelength band for light transmission, the second wavelength band beinga predetermined range having a lower limit and an upper limit within thefirst wavelength band, and the second wavelength band being set to beincluded in the third wavelength band of the color filter, the methodcomprising: an imaging step at which an optical signal inputted throughthe optical filter and the color filter is converted into a sensorsignal in the imaging sensor; a first signal processing step at which aninfrared light component is removed from the sensor signal outputted atthe imaging step; and a second signal processing step at which aninfrared image output signal is outputted from the sensor signaloutputted at the imaging step.